agora inbox for [email protected]
help / color / mirror / Atom feedRe: Allow COPY's 'text' format to output a header
65+ messages / 12 participants
[nested] [flat]
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-01 14:20 Daniel Verite <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Daniel Verite @ 2018-08-01 14:20 UTC (permalink / raw)
To: Simon Muller <[email protected]>; +Cc: Cynthia Shang <[email protected]>; pgsql-hackers
Simon Muller wrote:
> I've incorporated both your suggestions and included the patch you provided
> in the attached patch. Hope it's as expected.
Still unconvinced about the use case, since COPY's text format is only
meant to be consumed by Postgres, and the only way that Postgres will
consume this header is to discard it (at least as of the current
patch). But anyway...
/* Check header */
- if (!cstate->csv_mode && cstate->header_line)
+ if (cstate->binary && cstate->header_line)
ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("COPY HEADER available only in CSV mode")));
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("cannot specify HEADER in BINARY mode")));
Why should ERRCODE_FEATURE_NOT_SUPPORTED become ERRCODE_SYNTAX_ERROR?
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-01 15:18 Cynthia Shang <[email protected]>
parent: Daniel Verite <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Cynthia Shang @ 2018-08-01 15:18 UTC (permalink / raw)
To: Daniel Verite <[email protected]>; +Cc: Simon Muller <[email protected]>; pgsql-hackers
> On Aug 1, 2018, at 10:20 AM, Daniel Verite <[email protected]> wrote:
>
> /* Check header */
> - if (!cstate->csv_mode && cstate->header_line)
> + if (cstate->binary && cstate->header_line)
> ereport(ERROR,
> - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> - errmsg("COPY HEADER available only in CSV mode")));
> + (errcode(ERRCODE_SYNTAX_ERROR),
> + errmsg("cannot specify HEADER in BINARY mode")));
>
> Why should ERRCODE_FEATURE_NOT_SUPPORTED become ERRCODE_SYNTAX_ERROR?
>
I agree; it should remain ERRCODE_FEATURE_NOT_SUPPORTED and I might also suggest the message read "COPY HEADER not available in BINARY mode", although I'm pretty agnostic on the latter.
Regards,
-Cynthia Shang
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-01 18:36 Simon Muller <[email protected]>
parent: Cynthia Shang <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Simon Muller @ 2018-08-01 18:36 UTC (permalink / raw)
To: Cynthia Shang <[email protected]>; +Cc: Daniel Verite <[email protected]>; pgsql-hackers
On 1 August 2018 at 17:18, Cynthia Shang <[email protected]>
wrote:
>
> > On Aug 1, 2018, at 10:20 AM, Daniel Verite <[email protected]>
> wrote:
> >
> > /* Check header */
> > - if (!cstate->csv_mode && cstate->header_line)
> > + if (cstate->binary && cstate->header_line)
> > ereport(ERROR,
> > - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
> > - errmsg("COPY HEADER available only in CSV mode")));
> > + (errcode(ERRCODE_SYNTAX_ERROR),
> > + errmsg("cannot specify HEADER in BINARY mode")));
> >
> > Why should ERRCODE_FEATURE_NOT_SUPPORTED become ERRCODE_SYNTAX_ERROR?
> >
>
> I agree; it should remain ERRCODE_FEATURE_NOT_SUPPORTED and I might also
> suggest the message read "COPY HEADER not available in BINARY mode",
> although I'm pretty agnostic on the latter.
>
> Regards,
> -Cynthia Shang
I changed the error type and message for consistency with other similar
errors in that file. Whenever options are combined that are incompatible,
it looks like the convention is for a ERRCODE_SYNTAX_ERROR to be thrown.
For instance, in case you both specify a specific DELIMITER but also
declare the format as BINARY, then there is this code in that same file:
if (cstate->binary && cstate->delim)
ereport(ERROR,
(errcode(ERRCODE_SYNTAX_ERROR),
errmsg("cannot specify DELIMITER in BINARY mode")));
HEADER seems very similar to me since, like DELIMITER, it makes sense for
the textual formats such as CSV and TEXT, but doesn't make sense with the
BINARY format.
ERRCODE_FEATURE_NOT_SUPPORTED previously made sense since the only reason
TEXT and HEADER weren't compatible options was because the feature was not
yet implemented, but now ERRCODE_SYNTAX_ERROR seems to make sense to me
since I can't foresee a use case where BINARY and HEADER would ever be
compatible options.
--
Simon Muller
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-02 12:11 Daniel Verite <[email protected]>
parent: Simon Muller <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Daniel Verite @ 2018-08-02 12:11 UTC (permalink / raw)
To: Simon Muller <[email protected]>; +Cc: Cynthia Shang <[email protected]>; pgsql-hackers
Simon Muller wrote:
> I changed the error type and message for consistency with other similar
> errors in that file. Whenever options are combined that are incompatible,
> it looks like the convention is for a ERRCODE_SYNTAX_ERROR to be thrown.
That makes sense, thanks for elaborating, although there are also
a fair number of ERRCODE_FEATURE_NOT_SUPPORTED in copy.c
that are raised on forbidden/nonsensical combination of features,
so the consistency argument could work both ways.
Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-02 15:07 Cynthia Shang <[email protected]>
parent: Daniel Verite <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Cynthia Shang @ 2018-08-02 15:07 UTC (permalink / raw)
To: Daniel Verite <[email protected]>; +Cc: Simon Muller <[email protected]>; pgsql-hackers
> On Aug 2, 2018, at 8:11 AM, Daniel Verite <[email protected]> wrote:
>
> That makes sense, thanks for elaborating, although there are also
> a fair number of ERRCODE_FEATURE_NOT_SUPPORTED in copy.c
> that are raised on forbidden/nonsensical combination of features,
> so the consistency argument could work both ways.
>
If there is not a strong reason to change the error code, then I believe we should not. The error is the same as it was before, just narrower in scope.
Best,
-Cynthia
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-02 19:30 Simon Muller <[email protected]>
parent: Cynthia Shang <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Simon Muller @ 2018-08-02 19:30 UTC (permalink / raw)
To: Cynthia Shang <[email protected]>; +Cc: Daniel Verite <[email protected]>; pgsql-hackers
On 2 August 2018 at 17:07, Cynthia Shang <[email protected]>
wrote:
>
> > On Aug 2, 2018, at 8:11 AM, Daniel Verite <[email protected]>
> wrote:
> >
> > That makes sense, thanks for elaborating, although there are also
> > a fair number of ERRCODE_FEATURE_NOT_SUPPORTED in copy.c
> > that are raised on forbidden/nonsensical combination of features,
> > so the consistency argument could work both ways.
> >
>
> If there is not a strong reason to change the error code, then I believe
> we should not. The error is the same as it was before, just narrower in
> scope.
>
> Best,
> -Cynthia
Sure, thanks both for the feedback. Attached is a patch with the error kept
as ERRCODE_FEATURE_NOT_SUPPORTED.
--
Simon Muller
Attachments:
[application/octet-stream] text_header_v5.patch (6.6K, ../../CAF1-J-0adWePZkETpea1PEtdGChT5yu0AegPgOnNNdQsnx1g3w@mail.gmail.com/3-text_header_v5.patch)
download | inline diff:
diff --git a/contrib/file_fdw/input/file_fdw.source b/contrib/file_fdw/input/file_fdw.source
index a5e79a4549..4c6bc24913 100644
--- a/contrib/file_fdw/input/file_fdw.source
+++ b/contrib/file_fdw/input/file_fdw.source
@@ -37,7 +37,6 @@ CREATE USER MAPPING FOR regress_no_priv_user SERVER file_server;
-- validator tests
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'xml'); -- ERROR
-CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', header 'true'); -- ERROR
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', quote ':'); -- ERROR
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', escape ':'); -- ERROR
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'binary', header 'true'); -- ERROR
diff --git a/contrib/file_fdw/output/file_fdw.source b/contrib/file_fdw/output/file_fdw.source
index 853c9f9b28..adecd10d2b 100644
--- a/contrib/file_fdw/output/file_fdw.source
+++ b/contrib/file_fdw/output/file_fdw.source
@@ -33,14 +33,12 @@ CREATE USER MAPPING FOR regress_no_priv_user SERVER file_server;
-- validator tests
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'xml'); -- ERROR
ERROR: COPY format "xml" not recognized
-CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', header 'true'); -- ERROR
-ERROR: COPY HEADER available only in CSV mode
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', quote ':'); -- ERROR
ERROR: COPY quote available only in CSV mode
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', escape ':'); -- ERROR
ERROR: COPY escape available only in CSV mode
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'binary', header 'true'); -- ERROR
-ERROR: COPY HEADER available only in CSV mode
+ERROR: cannot specify HEADER in BINARY mode
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'binary', quote ':'); -- ERROR
ERROR: COPY quote available only in CSV mode
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'binary', escape ':'); -- ERROR
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 13a8b68d95..4db97589fd 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -279,7 +279,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
Specifies that the file contains a header line with the names of each
column in the file. On output, the first line contains the column
names from the table, and on input, the first line is ignored.
- This option is allowed only when using <literal>CSV</literal> format.
+ This option is not allowed when using <literal>binary</literal> format.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 3a66cb5025..fe9e25b988 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -1293,10 +1293,10 @@ ProcessCopyOptions(ParseState *pstate,
errmsg("COPY delimiter cannot be \"%s\"", cstate->delim)));
/* Check header */
- if (!cstate->csv_mode && cstate->header_line)
+ if (cstate->binary && cstate->header_line)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("COPY HEADER available only in CSV mode")));
+ errmsg("cannot specify HEADER in BINARY mode")));
/* Check quote */
if (!cstate->csv_mode && cstate->quote != NULL)
@@ -2033,8 +2033,11 @@ CopyTo(CopyState cstate)
colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname);
- CopyAttributeOutCSV(cstate, colname, false,
+ if (cstate->csv_mode)
+ CopyAttributeOutCSV(cstate, colname, false,
list_length(cstate->attnumlist) == 1);
+ else
+ CopyAttributeOutText(cstate, colname);
}
CopySendEndOfRow(cstate);
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index eb9e4b9774..f7cd364b63 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -557,6 +557,23 @@ SELECT * FROM instead_of_insert_tbl;
1 | test1
(1 row)
+-- test header line feature
+create temp table copytest3 (
+ c1 int,
+ "col with , comma" text,
+ "col with "" quote" int);
+copy copytest3 from stdin csv header;
+copy copytest3 to stdout csv header;
+c1,"col with , comma","col with "" quote"
+1,a,1
+2,b,2
+copy copytest3 from stdin header;
+copy copytest3 to stdout with (format text, header true);
+c1 col with , comma col with " quote
+1 a 1
+2 b 2
+3 c 3
+4 d 4
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
diff --git a/src/test/regress/input/copy.source b/src/test/regress/input/copy.source
index cb13606d14..20a140ab78 100644
--- a/src/test/regress/input/copy.source
+++ b/src/test/regress/input/copy.source
@@ -117,19 +117,3 @@ copy copytest to '@abs_builddir@/results/copytest.csv' csv quote '''' escape E'\
copy copytest2 from '@abs_builddir@/results/copytest.csv' csv quote '''' escape E'\\';
select * from copytest except select * from copytest2;
-
-
--- test header line feature
-
-create temp table copytest3 (
- c1 int,
- "col with , comma" text,
- "col with "" quote" int);
-
-copy copytest3 from stdin csv header;
-this is just a line full of junk that would error out if parsed
-1,a,1
-2,b,2
-\.
-
-copy copytest3 to stdout csv header;
diff --git a/src/test/regress/output/copy.source b/src/test/regress/output/copy.source
index b7e372d61b..9314622768 100644
--- a/src/test/regress/output/copy.source
+++ b/src/test/regress/output/copy.source
@@ -85,13 +85,3 @@ select * from copytest except select * from copytest2;
-------+------+--------
(0 rows)
--- test header line feature
-create temp table copytest3 (
- c1 int,
- "col with , comma" text,
- "col with "" quote" int);
-copy copytest3 from stdin csv header;
-copy copytest3 to stdout csv header;
-c1,"col with , comma","col with "" quote"
-1,a,1
-2,b,2
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index f3a6d228fa..c74a1e6469 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -411,6 +411,27 @@ test1
SELECT * FROM instead_of_insert_tbl;
+-- test header line feature
+create temp table copytest3 (
+ c1 int,
+ "col with , comma" text,
+ "col with "" quote" int);
+
+copy copytest3 from stdin csv header;
+this is just a line full of junk that would error out if parsed
+1,a,1
+2,b,2
+\.
+
+copy copytest3 to stdout csv header;
+
+copy copytest3 from stdin header;
+this is just a line full of junk that would error out if parsed
+3 c 3
+4 d 4
+\.
+
+copy copytest3 to stdout with (format text, header true);
-- clean up
DROP TABLE forcetest;
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-03 16:30 Cynthia Shang <[email protected]>
parent: Simon Muller <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Cynthia Shang @ 2018-08-03 16:30 UTC (permalink / raw)
To: Simon Muller <[email protected]>; +Cc: Daniel Verite <[email protected]>; pgsql-hackers
> On Aug 2, 2018, at 3:30 PM, Simon Muller <[email protected]> wrote:
>
> Sure, thanks both for the feedback. Attached is a patch with the error kept as ERRCODE_FEATURE_NOT_SUPPORTED.
>
I was able to apply the patch (after resolving a merge conflict which was expected given an update in master). All looks good.
-Cynthia
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-06 14:34 Stephen Frost <[email protected]>
parent: Cynthia Shang <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Stephen Frost @ 2018-08-06 14:34 UTC (permalink / raw)
To: Cynthia Shang <[email protected]>; +Cc: Simon Muller <[email protected]>; Daniel Verite <[email protected]>; pgsql-hackers
Greetings,
* Cynthia Shang ([email protected]) wrote:
> > On Aug 2, 2018, at 3:30 PM, Simon Muller <[email protected]> wrote:
> >
> > Sure, thanks both for the feedback. Attached is a patch with the error kept as ERRCODE_FEATURE_NOT_SUPPORTED.
>
> I was able to apply the patch (after resolving a merge conflict which was expected given an update in master). All looks good.
If there's a merge conflict against master, then it'd be good for an
updated patch to be posted.
Thanks!
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-08 18:57 Simon Muller <[email protected]>
parent: Stephen Frost <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Simon Muller @ 2018-08-08 18:57 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Cynthia Shang <[email protected]>; Daniel Verite <[email protected]>; pgsql-hackers
On 6 August 2018 at 16:34, Stephen Frost <[email protected]> wrote:
> Greetings,
>
> * Cynthia Shang ([email protected]) wrote:
> > I was able to apply the patch (after resolving a merge conflict which
> was expected given an update in master). All looks good.
>
> If there's a merge conflict against master, then it'd be good for an
> updated patch to be posted.
>
> Thanks!
>
> Stephen
>
Attached is an updated patch that should directly apply against current
master.
--
Simon Muller
Attachments:
[application/octet-stream] text_header_v6.patch (6.7K, ../../CAF1-J-0sm0wSUjhXdScL+QXyusnw5CmpG2XcYUPM5wyWZ9uQLQ@mail.gmail.com/3-text_header_v6.patch)
download | inline diff:
diff --git a/contrib/file_fdw/input/file_fdw.source b/contrib/file_fdw/input/file_fdw.source
index a5e79a4549..4c6bc24913 100644
--- a/contrib/file_fdw/input/file_fdw.source
+++ b/contrib/file_fdw/input/file_fdw.source
@@ -37,7 +37,6 @@ CREATE USER MAPPING FOR regress_no_priv_user SERVER file_server;
-- validator tests
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'xml'); -- ERROR
-CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', header 'true'); -- ERROR
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', quote ':'); -- ERROR
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', escape ':'); -- ERROR
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'binary', header 'true'); -- ERROR
diff --git a/contrib/file_fdw/output/file_fdw.source b/contrib/file_fdw/output/file_fdw.source
index 853c9f9b28..adecd10d2b 100644
--- a/contrib/file_fdw/output/file_fdw.source
+++ b/contrib/file_fdw/output/file_fdw.source
@@ -33,14 +33,12 @@ CREATE USER MAPPING FOR regress_no_priv_user SERVER file_server;
-- validator tests
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'xml'); -- ERROR
ERROR: COPY format "xml" not recognized
-CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', header 'true'); -- ERROR
-ERROR: COPY HEADER available only in CSV mode
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', quote ':'); -- ERROR
ERROR: COPY quote available only in CSV mode
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'text', escape ':'); -- ERROR
ERROR: COPY escape available only in CSV mode
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'binary', header 'true'); -- ERROR
-ERROR: COPY HEADER available only in CSV mode
+ERROR: cannot specify HEADER in BINARY mode
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'binary', quote ':'); -- ERROR
ERROR: COPY quote available only in CSV mode
CREATE FOREIGN TABLE tbl () SERVER file_server OPTIONS (format 'binary', escape ':'); -- ERROR
diff --git a/doc/src/sgml/ref/copy.sgml b/doc/src/sgml/ref/copy.sgml
index 13a8b68d95..4db97589fd 100644
--- a/doc/src/sgml/ref/copy.sgml
+++ b/doc/src/sgml/ref/copy.sgml
@@ -279,7 +279,7 @@ COPY { <replaceable class="parameter">table_name</replaceable> [ ( <replaceable
Specifies that the file contains a header line with the names of each
column in the file. On output, the first line contains the column
names from the table, and on input, the first line is ignored.
- This option is allowed only when using <literal>CSV</literal> format.
+ This option is not allowed when using <literal>binary</literal> format.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/commands/copy.c b/src/backend/commands/copy.c
index 9bc67ce60f..165883165d 100644
--- a/src/backend/commands/copy.c
+++ b/src/backend/commands/copy.c
@@ -1301,10 +1301,10 @@ ProcessCopyOptions(ParseState *pstate,
errmsg("COPY delimiter cannot be \"%s\"", cstate->delim)));
/* Check header */
- if (!cstate->csv_mode && cstate->header_line)
+ if (cstate->binary && cstate->header_line)
ereport(ERROR,
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("COPY HEADER available only in CSV mode")));
+ errmsg("cannot specify HEADER in BINARY mode")));
/* Check quote */
if (!cstate->csv_mode && cstate->quote != NULL)
@@ -2041,8 +2041,11 @@ CopyTo(CopyState cstate)
colname = NameStr(TupleDescAttr(tupDesc, attnum - 1)->attname);
- CopyAttributeOutCSV(cstate, colname, false,
+ if (cstate->csv_mode)
+ CopyAttributeOutCSV(cstate, colname, false,
list_length(cstate->attnumlist) == 1);
+ else
+ CopyAttributeOutText(cstate, colname);
}
CopySendEndOfRow(cstate);
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index eb9e4b9774..f7cd364b63 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -557,6 +557,23 @@ SELECT * FROM instead_of_insert_tbl;
1 | test1
(1 row)
+-- test header line feature
+create temp table copytest3 (
+ c1 int,
+ "col with , comma" text,
+ "col with "" quote" int);
+copy copytest3 from stdin csv header;
+copy copytest3 to stdout csv header;
+c1,"col with , comma","col with "" quote"
+1,a,1
+2,b,2
+copy copytest3 from stdin header;
+copy copytest3 to stdout with (format text, header true);
+c1 col with , comma col with " quote
+1 a 1
+2 b 2
+3 c 3
+4 d 4
-- clean up
DROP TABLE forcetest;
DROP TABLE vistest;
diff --git a/src/test/regress/input/copy.source b/src/test/regress/input/copy.source
index 014b1b5711..dae1b43f4b 100644
--- a/src/test/regress/input/copy.source
+++ b/src/test/regress/input/copy.source
@@ -118,22 +118,6 @@ copy copytest2 from '@abs_builddir@/results/copytest.csv' csv quote '''' escape
select * from copytest except select * from copytest2;
-
--- test header line feature
-
-create temp table copytest3 (
- c1 int,
- "col with , comma" text,
- "col with "" quote" int);
-
-copy copytest3 from stdin csv header;
-this is just a line full of junk that would error out if parsed
-1,a,1
-2,b,2
-\.
-
-copy copytest3 to stdout csv header;
-
-- test copy from with a partitioned table
create table parted_copytest (
a int,
diff --git a/src/test/regress/output/copy.source b/src/test/regress/output/copy.source
index ab096153ad..1c58908ae5 100644
--- a/src/test/regress/output/copy.source
+++ b/src/test/regress/output/copy.source
@@ -85,16 +85,6 @@ select * from copytest except select * from copytest2;
-------+------+--------
(0 rows)
--- test header line feature
-create temp table copytest3 (
- c1 int,
- "col with , comma" text,
- "col with "" quote" int);
-copy copytest3 from stdin csv header;
-copy copytest3 to stdout csv header;
-c1,"col with , comma","col with "" quote"
-1,a,1
-2,b,2
-- test copy from with a partitioned table
create table parted_copytest (
a int,
diff --git a/src/test/regress/sql/copy2.sql b/src/test/regress/sql/copy2.sql
index f3a6d228fa..c74a1e6469 100644
--- a/src/test/regress/sql/copy2.sql
+++ b/src/test/regress/sql/copy2.sql
@@ -411,6 +411,27 @@ test1
SELECT * FROM instead_of_insert_tbl;
+-- test header line feature
+create temp table copytest3 (
+ c1 int,
+ "col with , comma" text,
+ "col with "" quote" int);
+
+copy copytest3 from stdin csv header;
+this is just a line full of junk that would error out if parsed
+1,a,1
+2,b,2
+\.
+
+copy copytest3 to stdout csv header;
+
+copy copytest3 from stdin header;
+this is just a line full of junk that would error out if parsed
+3 c 3
+4 d 4
+\.
+
+copy copytest3 to stdout with (format text, header true);
-- clean up
DROP TABLE forcetest;
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-09 14:37 Cynthia Shang <[email protected]>
parent: Simon Muller <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Cynthia Shang @ 2018-08-09 14:37 UTC (permalink / raw)
To: Simon Muller <[email protected]>; +Cc: Stephen Frost <[email protected]>; Daniel Verite <[email protected]>; pgsql-hackers
> On Aug 8, 2018, at 2:57 PM, Simon Muller <[email protected]> wrote:
>
> If there's a merge conflict against master, then it'd be good for an
> updated patch to be posted.
>
> Thanks!
>
> Stephen
>
> Attached is an updated patch that should directly apply against current master.
>
> --
> Simon Muller
>
> <text_header_v6.patch>
This patch looks good. I realized I should have changed the status back while we were discussing all this. It is now (and still is) ready for committer.
Thanks,
-Cynthia
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Allow COPY's 'text' format to output a header
@ 2018-08-17 04:39 Michael Paquier <[email protected]>
parent: Cynthia Shang <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Michael Paquier @ 2018-08-17 04:39 UTC (permalink / raw)
To: Cynthia Shang <[email protected]>; +Cc: Simon Muller <[email protected]>; Stephen Frost <[email protected]>; Daniel Verite <[email protected]>; pgsql-hackers
On Thu, Aug 09, 2018 at 10:37:28AM -0400, Cynthia Shang wrote:
> This patch looks good. I realized I should have changed the status
> back while we were discussing all this. It is now (and still is) ready
> for committer.
I have some comments.
-ERROR: COPY HEADER available only in CSV mode
+ERROR: cannot specify HEADER in BINARY mode
This should read "COPY HEADER not available in BINARY mode" perhaps?
+copy copytest3 from stdin csv header;
+copy copytest3 to stdout csv header;
It would be more interesting to first export the data into the file with
a header, truncate the relation, and import it back with again header
specified. The data of the original should match the new, for both text
and csv format.
CopyStateData defines header_line, which still assumes that only CSV is
supported.
Why are there no additional tests for file_fdw?
The point about the header matching mentioned upthread is quite
interesting as it could make the proposed feature way more useful, and
it has not really been discussed. As far as I can see this adds more
sanity checks in NextCopyFromRawFields(). I'd like to think that this
should be a completely different option, say CHECK_HEADER, as CSV simply
skips the header in COPY FROM if specified on HEAD.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH v5] make \d pg_toast.foo show its indices
@ 2019-05-03 14:24 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2019-05-03 14:24 UTC (permalink / raw)
---
src/bin/psql/describe.c | 1 +
src/test/regress/expected/psql.out | 2 ++
2 files changed, 3 insertions(+)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 13ed2e1..86a7610 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2303,6 +2303,7 @@ describeOneTableDetails(const char *schemaname,
else if (tableinfo.relkind == RELKIND_RELATION ||
tableinfo.relkind == RELKIND_MATVIEW ||
tableinfo.relkind == RELKIND_FOREIGN_TABLE ||
+ tableinfo.relkind == RELKIND_TOASTVALUE ||
tableinfo.relkind == RELKIND_PARTITIONED_TABLE)
{
/* Footer information about a table */
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 5c8e439..d53dbb0 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -4757,4 +4757,6 @@ TOAST table "pg_toast.pg_toast_2619"
chunk_seq | integer
chunk_data | bytea
For table: "pg_catalog.pg_statistic"
+Indexes:
+ "pg_toast_2619_index" PRIMARY KEY, btree (chunk_id, chunk_seq)
--
2.7.4
--dkEUBIird37B8yKS
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v5-0003-show-childs-of-partitioned-indices.patch"
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH v2 1/2] make \d pg_toast.foo show its indices
@ 2019-05-03 14:24 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2019-05-03 14:24 UTC (permalink / raw)
---
src/bin/psql/describe.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index d7390d5..d26d986 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2274,6 +2274,7 @@ describeOneTableDetails(const char *schemaname,
else if (tableinfo.relkind == RELKIND_RELATION ||
tableinfo.relkind == RELKIND_MATVIEW ||
tableinfo.relkind == RELKIND_FOREIGN_TABLE ||
+ tableinfo.relkind == RELKIND_TOASTVALUE ||
tableinfo.relkind == RELKIND_PARTITIONED_TABLE)
{
/* Footer information about a table */
--
2.7.4
--brEuL7wsLY8+TuWz
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v2-0002-print-table-associated-with-given-TOAST-table.patch"
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH v3] make \d pg_toast.foo show its indices
@ 2019-05-03 14:24 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2019-05-03 14:24 UTC (permalink / raw)
---
src/bin/psql/describe.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index af2f440..c65bc82 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2275,6 +2275,7 @@ describeOneTableDetails(const char *schemaname,
else if (tableinfo.relkind == RELKIND_RELATION ||
tableinfo.relkind == RELKIND_MATVIEW ||
tableinfo.relkind == RELKIND_FOREIGN_TABLE ||
+ tableinfo.relkind == RELKIND_TOASTVALUE ||
tableinfo.relkind == RELKIND_PARTITIONED_TABLE)
{
/* Footer information about a table */
--
2.7.4
--AqsLC8rIMeq19msA
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v4-0002-show-childs-and-tablespaces-of-partitioned-indice.patch"
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH v3] make \d pg_toast.foo show its indices
@ 2019-05-03 14:24 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2019-05-03 14:24 UTC (permalink / raw)
---
src/bin/psql/describe.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index af2f440..c65bc82 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2275,6 +2275,7 @@ describeOneTableDetails(const char *schemaname,
else if (tableinfo.relkind == RELKIND_RELATION ||
tableinfo.relkind == RELKIND_MATVIEW ||
tableinfo.relkind == RELKIND_FOREIGN_TABLE ||
+ tableinfo.relkind == RELKIND_TOASTVALUE ||
tableinfo.relkind == RELKIND_PARTITIONED_TABLE)
{
/* Footer information about a table */
--
2.7.4
--LpQ9ahxlCli8rRTG
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v3-0002-show-childs-of-partitioned-indices.patch"
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH v7] make \d pg_toast.foo show its indices
@ 2019-05-03 14:24 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2019-05-03 14:24 UTC (permalink / raw)
---
src/bin/psql/describe.c | 1 +
src/test/regress/expected/psql.out | 2 ++
2 files changed, 3 insertions(+)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 9cd2e7d..b3b94d1 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2304,6 +2304,7 @@ describeOneTableDetails(const char *schemaname,
else if (tableinfo.relkind == RELKIND_RELATION ||
tableinfo.relkind == RELKIND_MATVIEW ||
tableinfo.relkind == RELKIND_FOREIGN_TABLE ||
+ tableinfo.relkind == RELKIND_TOASTVALUE ||
tableinfo.relkind == RELKIND_PARTITIONED_TABLE)
{
/* Footer information about a table */
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 5c8e439..d53dbb0 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -4757,4 +4757,6 @@ TOAST table "pg_toast.pg_toast_2619"
chunk_seq | integer
chunk_data | bytea
For table: "pg_catalog.pg_statistic"
+Indexes:
+ "pg_toast_2619_index" PRIMARY KEY, btree (chunk_id, chunk_seq)
--
2.7.4
--ryJZkp9/svQ58syV
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v7-0003-show-childs-of-partitioned-indices.patch"
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH v5] make \d pg_toast.foo show its indices
@ 2019-05-03 14:24 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2019-05-03 14:24 UTC (permalink / raw)
---
src/bin/psql/describe.c | 1 +
src/test/regress/expected/psql.out | 2 ++
2 files changed, 3 insertions(+)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 13ed2e1..86a7610 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -2303,6 +2303,7 @@ describeOneTableDetails(const char *schemaname,
else if (tableinfo.relkind == RELKIND_RELATION ||
tableinfo.relkind == RELKIND_MATVIEW ||
tableinfo.relkind == RELKIND_FOREIGN_TABLE ||
+ tableinfo.relkind == RELKIND_TOASTVALUE ||
tableinfo.relkind == RELKIND_PARTITIONED_TABLE)
{
/* Footer information about a table */
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 5c8e439..d53dbb0 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -4757,4 +4757,6 @@ TOAST table "pg_toast.pg_toast_2619"
chunk_seq | integer
chunk_data | bytea
For table: "pg_catalog.pg_statistic"
+Indexes:
+ "pg_toast_2619_index" PRIMARY KEY, btree (chunk_id, chunk_seq)
--
2.7.4
--Qxx1br4bt0+wmkIi
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v6-0003-show-childs-of-partitioned-indices.patch"
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH 4/4] Move the double-plus "Size" columns to the right
@ 2021-12-17 15:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2021-12-17 15:35 UTC (permalink / raw)
\dn, \dA, \db, \l, \d and \dP+
It doesn't make much sense that one cannot show a database's default tablespace
without also showing its size, and stat()ing every segment of every relation
in the DB.
---
src/bin/psql/describe.c | 42 ++++++++++++++---------
src/test/regress/expected/psql.out | 54 +++++++++++++++---------------
2 files changed, 53 insertions(+), 43 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ee7990ab7e4..44ae6ca9a75 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -224,11 +224,6 @@ describeTablespaces(const char *pattern, int verbose)
appendPQExpBufferStr(&buf, ",\n ");
printACLColumn(&buf, "spcacl");
- if (verbose > 1)
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
- gettext_noop("Size"));
-
appendPQExpBuffer(&buf,
",\n spcoptions AS \"%s\""
",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
@@ -236,6 +231,11 @@ describeTablespaces(const char *pattern, int verbose)
gettext_noop("Description"));
}
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_tablespace\n");
@@ -907,17 +907,22 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("Ctype"));
appendPQExpBufferStr(&buf, " ");
printACLColumn(&buf, "d.datacl");
- if (verbose > 1)
+
+ if (verbose > 0)
appendPQExpBuffer(&buf,
- ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
- " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
- " ELSE 'No Access'\n"
- " END as \"%s\""
",\n t.spcname as \"%s\""
",\n pg_catalog.shobj_description(d.oid, 'pg_database') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Tablespace"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
+ " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
+ " ELSE 'No Access'\n"
+ " END as \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_database d\n");
if (verbose > 0)
@@ -3780,10 +3785,13 @@ listTables(const char *tabtypes, const char *pattern, int verbose, bool showSyst
gettext_noop("Access method"));
appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\""
",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\"",
+ gettext_noop("Size"));
}
appendPQExpBufferStr(&buf,
@@ -3962,6 +3970,11 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
gettext_noop("Table"));
if (verbose > 0)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
+ gettext_noop("Description"));
+
+ if (verbose > 1)
{
if (showNested)
{
@@ -3978,9 +3991,6 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
",\n s.tps as \"%s\"",
gettext_noop("Total size"));
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 6428ebc507d..47166727318 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -2857,47 +2857,47 @@ Access method: heap
-- AM is displayed for tables, indexes and materialized views.
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent | |
(4 rows)
\dt+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+---------------+-------+----------------------+-------------+---------------+---------+-------------
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+---------------+-------+----------------------+-------------+---------------+-------------
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
(2 rows)
\dm+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
(1 row)
-- But not for views and sequences.
\dv+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+----------------+------+----------------------+-------------+---------+-------------
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+----------------+------+----------------------+-------------+-------------
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(1 row)
\set HIDE_TABLEAM on
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+--------------------+-------------------+----------------------+-------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent |
+ tableam_display | tbl_heap | table | regress_display_role | permanent |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(4 rows)
RESET ROLE;
--
2.17.1
--XLWMkxR+mZNQ4WTO--
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH v4 4/4] Move the double-plus "Size" columns to the right
@ 2021-12-17 15:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2021-12-17 15:35 UTC (permalink / raw)
\dn, \dA, \db, \l, \d and \dP+
It doesn't make much sense that one cannot show a database's default tablespace
without also showing its size, and stat()ing every segment of every relation
in the DB.
---
src/bin/psql/describe.c | 42 ++++++++++++++---------
src/test/regress/expected/psql.out | 54 +++++++++++++++---------------
2 files changed, 53 insertions(+), 43 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 710bf8ef91..42bad9281e 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -224,11 +224,6 @@ describeTablespaces(const char *pattern, int verbose)
appendPQExpBufferStr(&buf, ",\n ");
printACLColumn(&buf, "spcacl");
- if (verbose > 1)
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
- gettext_noop("Size"));
-
appendPQExpBuffer(&buf,
",\n spcoptions AS \"%s\""
",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
@@ -236,6 +231,11 @@ describeTablespaces(const char *pattern, int verbose)
gettext_noop("Description"));
}
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_tablespace\n");
@@ -907,17 +907,22 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("Ctype"));
appendPQExpBufferStr(&buf, " ");
printACLColumn(&buf, "d.datacl");
- if (verbose > 1)
+
+ if (verbose > 0)
appendPQExpBuffer(&buf,
- ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
- " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
- " ELSE 'No Access'\n"
- " END as \"%s\""
",\n t.spcname as \"%s\""
",\n pg_catalog.shobj_description(d.oid, 'pg_database') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Tablespace"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
+ " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
+ " ELSE 'No Access'\n"
+ " END as \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_database d\n");
if (verbose > 0)
@@ -3772,10 +3777,13 @@ listTables(const char *tabtypes, const char *pattern, int verbose, bool showSyst
gettext_noop("Access method"));
appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\""
",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\"",
+ gettext_noop("Size"));
}
appendPQExpBufferStr(&buf,
@@ -3954,6 +3962,11 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
gettext_noop("Table"));
if (verbose > 0)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
+ gettext_noop("Description"));
+
+ if (verbose > 1)
{
if (showNested)
{
@@ -3970,9 +3983,6 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
",\n s.tps as \"%s\"",
gettext_noop("Total size"));
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 930ce8597a..9a521a80b3 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -2845,47 +2845,47 @@ Access method: heap
-- AM is displayed for tables, indexes and materialized views.
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent | |
(4 rows)
\dt+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+---------------+-------+----------------------+-------------+---------------+---------+-------------
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+---------------+-------+----------------------+-------------+---------------+-------------
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
(2 rows)
\dm+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
(1 row)
-- But not for views and sequences.
\dv+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+----------------+------+----------------------+-------------+---------+-------------
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+----------------+------+----------------------+-------------+-------------
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(1 row)
\set HIDE_TABLEAM on
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+--------------------+-------------------+----------------------+-------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent |
+ tableam_display | tbl_heap | table | regress_display_role | permanent |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(4 rows)
RESET ROLE;
--
2.17.0
--2fEWJT3hVM9yyfvd--
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH 4/4] Move the double-plus "Size" columns to the right
@ 2021-12-17 15:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2021-12-17 15:35 UTC (permalink / raw)
\dn, \dA, \db, \l, \d and \dP+
It doesn't make much sense that one cannot show a database's default tablespace
without also showing its size, and stat()ing every segment of every relation
in the DB.
---
src/bin/psql/describe.c | 42 ++++++++++++++---------
src/test/regress/expected/psql.out | 54 +++++++++++++++---------------
2 files changed, 53 insertions(+), 43 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 220262aa84c..9f73e90df23 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -234,11 +234,6 @@ describeTablespaces(const char *pattern, int verbose)
appendPQExpBufferStr(&buf, ",\n ");
printACLColumn(&buf, "spcacl");
- if (verbose > 1)
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
- gettext_noop("Size"));
-
appendPQExpBuffer(&buf,
",\n spcoptions AS \"%s\""
",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
@@ -246,6 +241,11 @@ describeTablespaces(const char *pattern, int verbose)
gettext_noop("Description"));
}
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_tablespace\n");
@@ -941,17 +941,22 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("Locale Provider"));
appendPQExpBufferStr(&buf, " ");
printACLColumn(&buf, "d.datacl");
- if (verbose > 1)
+
+ if (verbose > 0)
appendPQExpBuffer(&buf,
- ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
- " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
- " ELSE 'No Access'\n"
- " END as \"%s\""
",\n t.spcname as \"%s\""
",\n pg_catalog.shobj_description(d.oid, 'pg_database') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Tablespace"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
+ " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
+ " ELSE 'No Access'\n"
+ " END as \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_database d\n");
if (verbose > 0)
@@ -3898,10 +3903,13 @@ listTables(const char *tabtypes, const char *pattern, int verbose, bool showSyst
gettext_noop("Access method"));
appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\""
",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\"",
+ gettext_noop("Size"));
}
appendPQExpBufferStr(&buf,
@@ -4082,6 +4090,11 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
gettext_noop("Table"));
if (verbose > 0)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
+ gettext_noop("Description"));
+
+ if (verbose > 1)
{
if (showNested)
{
@@ -4098,9 +4111,6 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
",\n s.tps as \"%s\"",
gettext_noop("Total size"));
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 60acbd1241e..8da23d7216c 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -2857,47 +2857,47 @@ Access method: heap
-- AM is displayed for tables, indexes and materialized views.
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent | |
(4 rows)
\dt+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+---------------+-------+----------------------+-------------+---------------+---------+-------------
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+---------------+-------+----------------------+-------------+---------------+-------------
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
(2 rows)
\dm+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
(1 row)
-- But not for views and sequences.
\dv+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+----------------+------+----------------------+-------------+---------+-------------
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+----------------+------+----------------------+-------------+-------------
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(1 row)
\set HIDE_TABLEAM on
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+--------------------+-------------------+----------------------+-------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent |
+ tableam_display | tbl_heap | table | regress_display_role | permanent |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(4 rows)
RESET ROLE;
--
2.17.1
--HcAYCG3uE/tztfnV--
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH v8 4/4] Move the double-plus "Size" columns to the right
@ 2021-12-17 15:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2021-12-17 15:35 UTC (permalink / raw)
\dn, \dA, \db, \l, \d and \dP+
It doesn't make much sense that one cannot show a database's default tablespace
without also showing its size, and stat()ing every segment of every relation
in the DB.
---
src/bin/psql/describe.c | 42 ++++++++++++++---------
src/test/regress/expected/psql.out | 54 +++++++++++++++---------------
2 files changed, 53 insertions(+), 43 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 2e47ea448f7..22fc19a47dd 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -240,11 +240,6 @@ describeTablespaces(const char *pattern, int verbose)
appendPQExpBufferStr(&buf, ",\n ");
printACLColumn(&buf, "spcacl");
- if (verbose > 1)
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
- gettext_noop("Size"));
-
appendPQExpBuffer(&buf,
",\n spcoptions AS \"%s\""
",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
@@ -252,6 +247,11 @@ describeTablespaces(const char *pattern, int verbose)
gettext_noop("Description"));
}
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_tablespace\n");
@@ -961,17 +961,22 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("Locale Provider"));
appendPQExpBufferStr(&buf, " ");
printACLColumn(&buf, "d.datacl");
- if (verbose > 1)
+
+ if (verbose > 0)
appendPQExpBuffer(&buf,
- ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
- " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
- " ELSE 'No Access'\n"
- " END as \"%s\""
",\n t.spcname as \"%s\""
",\n pg_catalog.shobj_description(d.oid, 'pg_database') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Tablespace"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
+ " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
+ " ELSE 'No Access'\n"
+ " END as \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_database d\n");
if (verbose > 0)
@@ -3936,10 +3941,13 @@ listTables(const char *tabtypes, const char *pattern, int verbose, bool showSyst
gettext_noop("Access method"));
appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\""
",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\"",
+ gettext_noop("Size"));
}
appendPQExpBufferStr(&buf,
@@ -4123,6 +4131,11 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
gettext_noop("Table"));
if (verbose > 0)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
+ gettext_noop("Description"));
+
+ if (verbose > 1)
{
if (showNested)
{
@@ -4139,9 +4152,6 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
",\n s.tps as \"%s\"",
gettext_noop("Total size"));
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index a7f5700edc1..2daa4685b4b 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -2858,47 +2858,47 @@ Access method: heap
-- AM is displayed for tables, indexes and materialized views.
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent | |
(4 rows)
\dt+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+---------------+-------+----------------------+-------------+---------------+---------+-------------
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+---------------+-------+----------------------+-------------+---------------+-------------
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
(2 rows)
\dm+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
(1 row)
-- But not for views and sequences.
\dv+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+----------------+------+----------------------+-------------+---------+-------------
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+----------------+------+----------------------+-------------+-------------
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(1 row)
\set HIDE_TABLEAM on
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+--------------------+-------------------+----------------------+-------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent |
+ tableam_display | tbl_heap | table | regress_display_role | permanent |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(4 rows)
RESET ROLE;
--
2.17.1
--cKDw3XFoqocuprIa--
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH 4/4] Move the double-plus "Size" columns to the right
@ 2021-12-17 15:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2021-12-17 15:35 UTC (permalink / raw)
\dn, \dA, \db, \l, and (for consistency) \d and \dP+
It doesn't make much sense that one cannot show a database's default
tablespace without also showing its size, and stat()ing every segment of
every relation in the DB.
---
src/bin/psql/describe.c | 40 ++++++++++++----------
src/test/regress/expected/psql.out | 54 +++++++++++++++---------------
2 files changed, 49 insertions(+), 45 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 5fc4f5774e8..a64e4ccba4d 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -241,17 +241,15 @@ describeTablespaces(const char *pattern, int verbose)
printACLColumn(&buf, "spcacl");
appendPQExpBuffer(&buf,
- ",\n spcoptions AS \"%s\"",
- gettext_noop("Options"));
+ ",\n spcoptions AS \"%s\""
+ ",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
+ gettext_noop("Options"),
+ gettext_noop("Description"));
if (verbose > 1)
appendPQExpBuffer(&buf,
",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
gettext_noop("Size"));
-
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
@@ -972,13 +970,6 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("ICU Rules"));
appendPQExpBufferStr(&buf, " ");
printACLColumn(&buf, "d.datacl");
- if (verbose > 1)
- appendPQExpBuffer(&buf,
- ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
- " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
- " ELSE 'No Access'\n"
- " END as \"%s\"",
- gettext_noop("Size"));
if (verbose > 0)
appendPQExpBuffer(&buf,
@@ -987,6 +978,14 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("Tablespace"),
gettext_noop("Description"));
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
+ " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
+ " ELSE 'No Access'\n"
+ " END as \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_database d\n");
if (verbose > 0)
@@ -3954,10 +3953,13 @@ listTables(const char *tabtypes, const char *pattern, int verbose, bool showSyst
gettext_noop("Access method"));
appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\""
",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\"",
+ gettext_noop("Size"));
}
appendPQExpBufferStr(&buf,
@@ -4141,6 +4143,11 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
gettext_noop("Table"));
if (verbose > 0)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
+ gettext_noop("Description"));
+
+ if (verbose > 1)
{
if (showNested)
{
@@ -4157,9 +4164,6 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
",\n s.tps as \"%s\"",
gettext_noop("Total size"));
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index c00e28361c7..b9c95502a45 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -2889,47 +2889,47 @@ Access method: heap
-- AM is displayed for tables, indexes and materialized views.
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent | |
(4 rows)
\dt+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+---------------+-------+----------------------+-------------+---------------+---------+-------------
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+---------------+-------+----------------------+-------------+---------------+-------------
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
(2 rows)
\dm+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
(1 row)
-- But not for views and sequences.
\dv+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+----------------+------+----------------------+-------------+---------+-------------
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+----------------+------+----------------------+-------------+-------------
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(1 row)
\set HIDE_TABLEAM on
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+--------------------+-------------------+----------------------+-------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent |
+ tableam_display | tbl_heap | table | regress_display_role | permanent |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(4 rows)
RESET ROLE;
--
2.34.1
--OZ1/qgOAlc2dTU8W--
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH 4/4] Move the double-plus "Size" columns to the right
@ 2021-12-17 15:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2021-12-17 15:35 UTC (permalink / raw)
\dn, \dA, \db, \l, \d and \dP+
It doesn't make much sense that one cannot show a database's default tablespace
without also showing its size, and stat()ing every segment of every relation
in the DB.
---
src/bin/psql/describe.c | 42 ++++++++++++++---------
src/test/regress/expected/psql.out | 54 +++++++++++++++---------------
2 files changed, 53 insertions(+), 43 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index f7b646062bd..77a883815fe 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -240,11 +240,6 @@ describeTablespaces(const char *pattern, int verbose)
appendPQExpBufferStr(&buf, ",\n ");
printACLColumn(&buf, "spcacl");
- if (verbose > 1)
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
- gettext_noop("Size"));
-
appendPQExpBuffer(&buf,
",\n spcoptions AS \"%s\""
",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
@@ -252,6 +247,11 @@ describeTablespaces(const char *pattern, int verbose)
gettext_noop("Description"));
}
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_tablespace\n");
@@ -961,17 +961,22 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("Locale Provider"));
appendPQExpBufferStr(&buf, " ");
printACLColumn(&buf, "d.datacl");
- if (verbose > 1)
+
+ if (verbose > 0)
appendPQExpBuffer(&buf,
- ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
- " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
- " ELSE 'No Access'\n"
- " END as \"%s\""
",\n t.spcname as \"%s\""
",\n pg_catalog.shobj_description(d.oid, 'pg_database') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Tablespace"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
+ " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
+ " ELSE 'No Access'\n"
+ " END as \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_database d\n");
if (verbose > 0)
@@ -3941,10 +3946,13 @@ listTables(const char *tabtypes, const char *pattern, int verbose, bool showSyst
gettext_noop("Access method"));
appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\""
",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\"",
+ gettext_noop("Size"));
}
appendPQExpBufferStr(&buf,
@@ -4128,6 +4136,11 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
gettext_noop("Table"));
if (verbose > 0)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
+ gettext_noop("Description"));
+
+ if (verbose > 1)
{
if (showNested)
{
@@ -4144,9 +4157,6 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
",\n s.tps as \"%s\"",
gettext_noop("Total size"));
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 8fc62cebd2d..cd95680994b 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -2889,47 +2889,47 @@ Access method: heap
-- AM is displayed for tables, indexes and materialized views.
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent | |
(4 rows)
\dt+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+---------------+-------+----------------------+-------------+---------------+---------+-------------
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+---------------+-------+----------------------+-------------+---------------+-------------
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
(2 rows)
\dm+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
(1 row)
-- But not for views and sequences.
\dv+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+----------------+------+----------------------+-------------+---------+-------------
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+----------------+------+----------------------+-------------+-------------
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(1 row)
\set HIDE_TABLEAM on
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+--------------------+-------------------+----------------------+-------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent |
+ tableam_display | tbl_heap | table | regress_display_role | permanent |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(4 rows)
RESET ROLE;
--
2.25.1
--H1spWtNR+x+ondvy--
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH 4/4] Move the double-plus "Size" columns to the right
@ 2021-12-17 15:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2021-12-17 15:35 UTC (permalink / raw)
\dn, \dA, \db, \l, and (for consistency) \d and \dP+
It doesn't make much sense that one cannot show a database's default
tablespace without also showing its size, and stat()ing every segment of
every relation in the DB.
---
src/bin/psql/describe.c | 40 ++++++++++++----------
src/test/regress/expected/psql.out | 54 +++++++++++++++---------------
2 files changed, 49 insertions(+), 45 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 8ab4b4cb46a..1b7e5c5bcd2 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -241,17 +241,15 @@ describeTablespaces(const char *pattern, int verbose)
printACLColumn(&buf, "spcacl");
appendPQExpBuffer(&buf,
- ",\n spcoptions AS \"%s\"",
- gettext_noop("Options"));
+ ",\n spcoptions AS \"%s\""
+ ",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
+ gettext_noop("Options"),
+ gettext_noop("Description"));
if (verbose > 1)
appendPQExpBuffer(&buf,
",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
gettext_noop("Size"));
-
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
@@ -972,13 +970,6 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("ICU Rules"));
appendPQExpBufferStr(&buf, " ");
printACLColumn(&buf, "d.datacl");
- if (verbose > 1)
- appendPQExpBuffer(&buf,
- ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
- " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
- " ELSE 'No Access'\n"
- " END as \"%s\"",
- gettext_noop("Size"));
if (verbose > 0)
appendPQExpBuffer(&buf,
@@ -987,6 +978,14 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("Tablespace"),
gettext_noop("Description"));
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
+ " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
+ " ELSE 'No Access'\n"
+ " END as \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_database d\n");
if (verbose > 0)
@@ -3951,10 +3950,13 @@ listTables(const char *tabtypes, const char *pattern, int verbose, bool showSyst
gettext_noop("Access method"));
appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\""
",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\"",
+ gettext_noop("Size"));
}
appendPQExpBufferStr(&buf,
@@ -4138,6 +4140,11 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
gettext_noop("Table"));
if (verbose > 0)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
+ gettext_noop("Description"));
+
+ if (verbose > 1)
{
if (showNested)
{
@@ -4154,9 +4161,6 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
",\n s.tps as \"%s\"",
gettext_noop("Total size"));
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 8f930283636..4666333f73f 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -2889,47 +2889,47 @@ Access method: heap
-- AM is displayed for tables, indexes and materialized views.
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent | |
(4 rows)
\dt+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+---------------+-------+----------------------+-------------+---------------+---------+-------------
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+---------------+-------+----------------------+-------------+---------------+-------------
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
(2 rows)
\dm+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
(1 row)
-- But not for views and sequences.
\dv+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+----------------+------+----------------------+-------------+---------+-------------
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+----------------+------+----------------------+-------------+-------------
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(1 row)
\set HIDE_TABLEAM on
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+--------------------+-------------------+----------------------+-------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent |
+ tableam_display | tbl_heap | table | regress_display_role | permanent |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(4 rows)
RESET ROLE;
--
2.34.1
--mDjj41GS8UJ6l/Dp--
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH 4/4] Move the double-plus "Size" columns to the right
@ 2021-12-17 15:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2021-12-17 15:35 UTC (permalink / raw)
\dn, \dA, \db, \l, and (for consistency) \d and \dP+
It doesn't make much sense that one cannot show a database's default
tablespace without also showing its size, and stat()ing every segment of
every relation in the DB.
---
src/bin/psql/describe.c | 40 ++++++++++++----------
src/test/regress/expected/psql.out | 54 +++++++++++++++---------------
2 files changed, 49 insertions(+), 45 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7eff1249015..cc5f00fffe7 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -241,17 +241,15 @@ describeTablespaces(const char *pattern, int verbose)
printACLColumn(&buf, "spcacl");
appendPQExpBuffer(&buf,
- ",\n spcoptions AS \"%s\"",
- gettext_noop("Options"));
+ ",\n spcoptions AS \"%s\""
+ ",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
+ gettext_noop("Options"),
+ gettext_noop("Description"));
if (verbose > 1)
appendPQExpBuffer(&buf,
",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
gettext_noop("Size"));
-
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
@@ -972,13 +970,6 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("ICU Rules"));
appendPQExpBufferStr(&buf, " ");
printACLColumn(&buf, "d.datacl");
- if (verbose > 1)
- appendPQExpBuffer(&buf,
- ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
- " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
- " ELSE 'No Access'\n"
- " END as \"%s\"",
- gettext_noop("Size"));
if (verbose > 0)
appendPQExpBuffer(&buf,
@@ -987,6 +978,14 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("Tablespace"),
gettext_noop("Description"));
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
+ " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
+ " ELSE 'No Access'\n"
+ " END as \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_database d\n");
if (verbose > 0)
@@ -4012,10 +4011,13 @@ listTables(const char *tabtypes, const char *pattern, int verbose, bool showSyst
gettext_noop("Access method"));
appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\""
",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\"",
+ gettext_noop("Size"));
}
appendPQExpBufferStr(&buf,
@@ -4199,6 +4201,11 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
gettext_noop("Table"));
if (verbose > 0)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
+ gettext_noop("Description"));
+
+ if (verbose > 1)
{
if (showNested)
{
@@ -4215,9 +4222,6 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
",\n s.tps as \"%s\"",
gettext_noop("Total size"));
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 7cd0c27cca8..e623a2df7c9 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -2900,47 +2900,47 @@ Access method: heap
-- AM is displayed for tables, indexes and materialized views.
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent | |
(4 rows)
\dt+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+---------------+-------+----------------------+-------------+---------------+---------+-------------
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+---------------+-------+----------------------+-------------+---------------+-------------
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
(2 rows)
\dm+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
(1 row)
-- But not for views and sequences.
\dv+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+----------------+------+----------------------+-------------+---------+-------------
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+----------------+------+----------------------+-------------+-------------
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(1 row)
\set HIDE_TABLEAM on
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+--------------------+-------------------+----------------------+-------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent |
+ tableam_display | tbl_heap | table | regress_display_role | permanent |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(4 rows)
RESET ROLE;
--
2.34.1
--X119r+cMzLJwahiZ--
^ permalink raw reply [nested|flat] 65+ messages in thread
* [PATCH 4/4] Move the double-plus "Size" columns to the right
@ 2021-12-17 15:35 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Justin Pryzby @ 2021-12-17 15:35 UTC (permalink / raw)
\dn, \dA, \db, \l, and (for consistency) \d and \dP+
It doesn't make much sense that one cannot show a database's default
tablespace without also showing its size, and stat()ing every segment of
every relation in the DB.
---
src/bin/psql/describe.c | 40 ++++++++++++----------
src/test/regress/expected/psql.out | 54 +++++++++++++++---------------
2 files changed, 49 insertions(+), 45 deletions(-)
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 7eff1249015..cc5f00fffe7 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -241,17 +241,15 @@ describeTablespaces(const char *pattern, int verbose)
printACLColumn(&buf, "spcacl");
appendPQExpBuffer(&buf,
- ",\n spcoptions AS \"%s\"",
- gettext_noop("Options"));
+ ",\n spcoptions AS \"%s\""
+ ",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
+ gettext_noop("Options"),
+ gettext_noop("Description"));
if (verbose > 1)
appendPQExpBuffer(&buf,
",\n pg_catalog.pg_size_pretty(pg_catalog.pg_tablespace_size(oid)) AS \"%s\"",
gettext_noop("Size"));
-
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.shobj_description(oid, 'pg_tablespace') AS \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
@@ -972,13 +970,6 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("ICU Rules"));
appendPQExpBufferStr(&buf, " ");
printACLColumn(&buf, "d.datacl");
- if (verbose > 1)
- appendPQExpBuffer(&buf,
- ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
- " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
- " ELSE 'No Access'\n"
- " END as \"%s\"",
- gettext_noop("Size"));
if (verbose > 0)
appendPQExpBuffer(&buf,
@@ -987,6 +978,14 @@ listAllDbs(const char *pattern, int verbose)
gettext_noop("Tablespace"),
gettext_noop("Description"));
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n CASE WHEN pg_catalog.has_database_privilege(d.datname, 'CONNECT')\n"
+ " THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname))\n"
+ " ELSE 'No Access'\n"
+ " END as \"%s\"",
+ gettext_noop("Size"));
+
appendPQExpBufferStr(&buf,
"\nFROM pg_catalog.pg_database d\n");
if (verbose > 0)
@@ -4012,10 +4011,13 @@ listTables(const char *tabtypes, const char *pattern, int verbose, bool showSyst
gettext_noop("Access method"));
appendPQExpBuffer(&buf,
- ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\""
",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Size"),
gettext_noop("Description"));
+
+ if (verbose > 1)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.pg_size_pretty(pg_catalog.pg_table_size(c.oid)) as \"%s\"",
+ gettext_noop("Size"));
}
appendPQExpBufferStr(&buf,
@@ -4199,6 +4201,11 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
gettext_noop("Table"));
if (verbose > 0)
+ appendPQExpBuffer(&buf,
+ ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
+ gettext_noop("Description"));
+
+ if (verbose > 1)
{
if (showNested)
{
@@ -4215,9 +4222,6 @@ listPartitionedTables(const char *reltypes, const char *pattern, int verbose)
",\n s.tps as \"%s\"",
gettext_noop("Total size"));
- appendPQExpBuffer(&buf,
- ",\n pg_catalog.obj_description(c.oid, 'pg_class') as \"%s\"",
- gettext_noop("Description"));
}
appendPQExpBufferStr(&buf,
diff --git a/src/test/regress/expected/psql.out b/src/test/regress/expected/psql.out
index 7cd0c27cca8..e623a2df7c9 100644
--- a/src/test/regress/expected/psql.out
+++ b/src/test/regress/expected/psql.out
@@ -2900,47 +2900,47 @@ Access method: heap
-- AM is displayed for tables, indexes and materialized views.
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent | |
(4 rows)
\dt+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+---------------+-------+----------------------+-------------+---------------+---------+-------------
- tableam_display | tbl_heap | table | regress_display_role | permanent | heap | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+---------------+-------+----------------------+-------------+---------------+-------------
+ tableam_display | tbl_heap | table | regress_display_role | permanent | heap |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent | heap_psql |
(2 rows)
\dm+
- List of relations
- Schema | Name | Type | Owner | Persistence | Access method | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Access method | Description
+-----------------+--------------------+-------------------+----------------------+-------------+---------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | heap_psql |
(1 row)
-- But not for views and sequences.
\dv+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+----------------+------+----------------------+-------------+---------+-------------
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+----------------+------+----------------------+-------------+-------------
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(1 row)
\set HIDE_TABLEAM on
\d+
- List of relations
- Schema | Name | Type | Owner | Persistence | Size | Description
------------------+--------------------+-------------------+----------------------+-------------+---------+-------------
- tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap | table | regress_display_role | permanent | 0 bytes |
- tableam_display | tbl_heap_psql | table | regress_display_role | permanent | 0 bytes |
- tableam_display | view_heap_psql | view | regress_display_role | permanent | 0 bytes |
+ List of relations
+ Schema | Name | Type | Owner | Persistence | Description
+-----------------+--------------------+-------------------+----------------------+-------------+-------------
+ tableam_display | mat_view_heap_psql | materialized view | regress_display_role | permanent |
+ tableam_display | tbl_heap | table | regress_display_role | permanent |
+ tableam_display | tbl_heap_psql | table | regress_display_role | permanent |
+ tableam_display | view_heap_psql | view | regress_display_role | permanent |
(4 rows)
RESET ROLE;
--
2.34.1
--X119r+cMzLJwahiZ--
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-04-30 10:43 Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Daniel Gustafsson @ 2025-04-30 10:43 UTC (permalink / raw)
To: Peter Eisentraut <[email protected]>; +Cc: Rahila Syed <[email protected]>; pgsql-hackers
> On 30 Apr 2025, at 12:14, Peter Eisentraut <[email protected]> wrote:
>
> On 29.04.25 15:13, Rahila Syed wrote:
>> Please find attached a patch with some comments and documentation changes.
>> Additionaly, added a missing '\0' termination to "Remaining Totals" string.
>> I think this became necessary after we replaced dsa_allocate0()
>> with dsa_allocate() is the latest version.
>
> > strncpy(nameptr, "Remaining Totals", namelen);
> > + nameptr[namelen] = '\0';
>
> Looks like a case for strlcpy()?
True. I did go ahead with the strncpy and nul terminator assignment, mostly
out of muscle memory, but I agree that this would be a good place for a
strlcpy() instead.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-07-11 15:31 Rahila Syed <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-07-11 15:31 UTC (permalink / raw)
To: pgsql-hackers
Hi,
Please find attached the latest memory context statistics monitoring patch.
It has been redesigned to address several issues highlighted in the thread
[1]
and [2].
Here are some key highlights of the new design:
- All DSA processing has been moved out of the CFI handler function. Now,
all the dynamic shared memory
needed to store the statistics is created and deleted in the client
function. This change addresses concerns
that DSA APIs are too high level to be safely called from interrupt
handlers. There was also a concern that
DSA API calls might not provide re-entrancy, which could cause issues if
CFI is invoked from a DSA function
in the future.
- The static shared memory array has been replaced with a DSHASH table
which now holds metadata such as
pointers to actual statistics for each process.
- dsm_registry.c APIs are used for creating and attaching to DSA and
DSHASH table, which helps prevent code
duplication.
-To address the memory leak concern, we create an exclusive memory context
under the NULL context, which
does not fall under the TopMemoryContext tree, to handle all the memory
allocations in ProcessGetMemoryContextInterrupt.
This ensures the memory context created by the function does not affect its
outcome.
The memory context is reset at the end of the function, which helps prevent
any memory leaks.
- Changes made to the mcxt.c file have been relocated to mcxtfuncs.c, which
now contains all the existing
memory statistics-related functions along with the code for the proposed
function.
The overall flow of a request is as follows:
1. A client backend running the pg_get_process_memory_contexts function
creates a DSA and allocates memory
to store statistics, tracked by DSA pointer. This pointer is stored in a
DSHASH entry for each client querying the
statistics of any process.
The client shares its DSHASH table key with the server process using a
static shared array of keys indexed
by the server's procNumber. It notifies the server process to publish
statistics by using SendProcSignal.
2. When a PostgreSQL server process handles the request for memory
statistics, the CFI function accesses the
client hash key stored in its procNumber slot of the shared keys array. The
server process then retrieves the
DSHASH entry to obtain the DSA pointer allocated by the client, for storing
the statistics.
After storing the statistics, it notifies the client through its condition
variable.
3. Although the DSA is created just once, the memory inside the DSA is
allocated and released by the client
process as soon as it finishes reading the statistics.
If it fails to do so, it is deleted by the before_shmem_exit callback when
the client exits. The client's entry in DSHASH
table is also deleted when the client exits.
4. The DSA and DSHASH table are not created
until pg_get_process_memory_context function is called.
Once created, any client backend querying statistics and any PostgreSQL
process publishing statistics will
attach to the same area and table.
Please let me know your thoughts.
Thank you,
Rahila Syed
[1]. PostgreSQL: Re: pgsql: Add function to get memory context stats for
processes
<https://www.postgresql.org/message-id/CA%2BTgmoaey-kOP1k5FaUnQFd1fR0majVebWcL8ogfLbG_nt-Ytg%40mail.g...;
[2]. PostgreSQL: Re: Prevent an error on attaching/creating a DSM/DSA from
an interrupt handler.
<https://www.postgresql.org/message-id/flat/8B873D49-E0E5-4F9F-B8D6-CA4836B825CD%40yesql.se#7026d2fe4...;
On Wed, Apr 30, 2025 at 4:13 PM Daniel Gustafsson <[email protected]> wrote:
> > On 30 Apr 2025, at 12:14, Peter Eisentraut <[email protected]> wrote:
> >
> > On 29.04.25 15:13, Rahila Syed wrote:
> >> Please find attached a patch with some comments and documentation
> changes.
> >> Additionaly, added a missing '\0' termination to "Remaining Totals"
> string.
> >> I think this became necessary after we replaced dsa_allocate0()
> >> with dsa_allocate() is the latest version.
> >
> > > strncpy(nameptr, "Remaining Totals", namelen);
> > > + nameptr[namelen] = '\0';
> >
> > Looks like a case for strlcpy()?
>
> True. I did go ahead with the strncpy and nul terminator assignment,
> mostly
> out of muscle memory, but I agree that this would be a good place for a
> strlcpy() instead.
>
> --
> Daniel Gustafsson
>
>
Attachments:
[application/octet-stream] v30-0001-Add-pg_get_process_memory_context-function.patch (60.0K, ../../CAH2L28sc-rEhyntPLoaC2XUa0ZjS5ka6KzEbuSVxQBBnUYu1KQ@mail.gmail.com/3-v30-0001-Add-pg_get_process_memory_context-function.patch)
download | inline diff:
From c4ea611583dd36c1a3facf7d3185c1e9e93b17b2 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Mon, 30 Jun 2025 12:11:00 +0530
Subject: [PATCH] Add pg_get_process_memory_context function
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes the caller specifies
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned.
---
doc/src/sgml/func.sgml | 164 ++++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/lwlock.c | 1 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 1 +
src/backend/utils/adt/mcxtfuncs.c | 837 +++++++++++++++++-
src/backend/utils/adt/pg_locale.c | 1 -
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mb/mbutils.c | 1 -
src/backend/utils/mmgr/mcxt.c | 71 +-
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlock.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 92 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 3 +
28 files changed, 1227 insertions(+), 41 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index c28aa71f570..ba082030ea2 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -28716,6 +28716,137 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres}
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type>, <parameter>timeout</parameter> <type>float</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type>,
+ <parameter>stats_timestamp</parameter> <type>timestamptz</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ Busy processes can delay reporting memory context statistics,
+ <parameter>timeout</parameter> specifies the number of seconds
+ to wait for updated statistics. <parameter>timeout</parameter> can be
+ specified in fractions of a second.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
@@ -28855,6 +28986,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false, 0.5) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+path | {1}
+level | 1
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index b2d5332effc..33b5fcb9119 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -682,6 +682,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 9474095f271..5e7e8081c05 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -781,6 +781,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fda91ffd1ce..d3cb3f1891c 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -663,6 +663,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index 0ae9bf906ec..f24f574e748 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..ac97a39447c 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -867,6 +867,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 777c9a8d555..5d14684f6b2 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 2fa045e6b0f..fe3d32e40b0 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -51,6 +51,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -150,6 +151,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -343,6 +345,7 @@ CreateOrAttachShmemStructs(void)
WaitEventCustomShmemInit();
InjectionPointShmemInit();
AioShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index a9bb540b55a..ce69e26d720 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 46f44bc4511..a7b5ede2b12 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -178,6 +178,7 @@ static const char *const BuiltinTrancheNames[] = {
[LWTRANCHE_XACT_SLRU] = "XactSLRU",
[LWTRANCHE_PARALLEL_VACUUM_DSA] = "ParallelVacuumDSA",
[LWTRANCHE_AIO_URING_COMPLETION] = "AioUringCompletion",
+ [LWTRANCHE_MEMORY_CONTEXT_KEYS] = "MemoryContextReportingKeys",
};
StaticAssertDecl(lengthof(BuiltinTrancheNames) ==
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..f194e6b3dcc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -50,6 +50,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2f8c3d5f918..83db8a20efb 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3533,6 +3533,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 4da68312b5f..78b1fa5ca43 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -161,6 +161,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..b44bcb18118 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,13 +15,38 @@
#include "postgres.h"
+#include "access/twophase.h"
+#include "catalog/pg_authid_d.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+#define CLIENT_KEY_SIZE 64
+
+static LWLock *client_keys_lock = NULL;
+static int *client_keys = NULL;
+static dshash_table *MemoryStatsDsHash = NULL;
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void dsa_cleanup(MemoryStatsDSHashEntry *entry);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts, int max_levels);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
@@ -89,7 +114,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +168,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +183,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +229,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +256,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +264,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +345,754 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to MEMSTATS_WAIT_TIMEOUT, retry given that there is
+ * time left within the timeout specified by the user, before giving up and
+ * returning previously published statistics, if any. If no previous statistics
+ * exist, return NULL.
+ */
+#define MEMSTATS_WAIT_TIMEOUT 100
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ double timeout = PG_GETARG_FLOAT8(2);
+ PGPROC *proc;
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit(1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ ereport(WARNING,
+ errmsg("server process is processing previous request %d: %m", pid));
+ LWLockRelease(client_keys_lock);
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ ConditionVariableInit(&entry->memcxt_cv);
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The meomry is freed at the end of the
+ * function.
+ */
+ Assert(!DsaPointerIsValid(entry->memstats_dsa_pointer));
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+ entry->summary = summary;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m", pid));
+ PG_RETURN_NULL();
+ }
+
+ while (1)
+ {
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using the correct
+ * entry in the proc_id field.
+ *
+ * Make sure that the information belongs to pid we requested
+ * information for, Otherwise loop back and wait for the server
+ * process to finish publishing statistics.
+ *
+ * Note in procnumber.h file says that a procNumber can be re-used for
+ * a different backend immediately after a backend exits. In case an
+ * old process' data was there and not updated by the current process
+ * in the slot identified by the procNumber, the pid of the requested
+ * process and the proc_id might not match.
+ *
+ */
+ if (entry->proc_id == pid)
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The process ending during memory context processing is not an
+ * error.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ dsa_cleanup(entry);
+ PG_RETURN_NULL();
+ }
+
+
+ /*
+ * Wait for the timeout as defined by the user. If no statistics are
+ * available within the allowed time then return NULL. The timer is
+ * defined in milliseconds since that's what the condition variable
+ * sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (timeout * 1000), WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ dsa_cleanup(entry);
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ if (memcxt_info[i].name[0] != '\0')
+ {
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+ }
+ else
+ nulls[0] = true;
+
+ if (memcxt_info[i].ident[0] != '\0')
+ {
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ }
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum, path_length, INT4OID);
+ values[3] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[3] = true;
+
+ values[4] = Int32GetDatum(memcxt_info[i].levels);
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dsa_cleanup(entry);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+dsa_cleanup(MemoryStatsDSHashEntry *entry)
+{
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->proc_id = 0;
+}
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Statistics written by each process are tracked independently in
+ * per-process DSA pointers. These pointers are stored in static shared memory.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility maximum size of statistics for each context. The remaining context
+ * statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ bool summary = false;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ /* Retreive the client key fo publishing statistics */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ LWLockRelease(client_keys_lock);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ summary = entry->summary;
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry->proc_id = MyProcPid;
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1, 100);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &c,
+ HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsInternal(c, 1, 100, 100, &grand_totals,
+ PRINT_STATS_NONE, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts, 100);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ /* Notify waiting backends and return */
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1, 100);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name, "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id -
+ num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+
+ /* Notify waiting backends and return */
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+}
+
+/*
+ * Update timestamp and signal all the waiting client backends after copying
+ * all the statistics.
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+
+ /*
+ * Empty this processes slot, so other clients can request memory
+ * statistics
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts, int max_levels)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), max_levels);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+}
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 97c2ac1faf9..ab768a7a91f 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -45,7 +45,6 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
-#include "utils/relcache.h"
#include "utils/syscache.h"
#ifdef WIN32
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index c86ceefda94..89d72cdd5ff 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -663,6 +663,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 886ecbad871..308016d7763 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -39,7 +39,6 @@
#include "mb/pg_wchar.h"
#include "utils/fmgrprotos.h"
#include "utils/memutils.h"
-#include "utils/relcache.h"
#include "varatt.h"
/*
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 15fa4d0a55e..d6b081d1bbf 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,7 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "utils/hsearch.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/memutils_internal.h"
@@ -159,10 +160,6 @@ MemoryContext PortalContext = NULL;
static void MemoryContextDeleteOnly(MemoryContext context);
static void MemoryContextCallResetCallbacks(MemoryContext context);
-static void MemoryContextStatsInternal(MemoryContext context, int level,
- int max_level, int max_children,
- MemoryContextCounters *totals,
- bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
const char *stats_string,
bool print_to_stderr);
@@ -831,11 +828,19 @@ MemoryContextStatsDetail(MemoryContext context,
bool print_to_stderr)
{
MemoryContextCounters grand_totals;
+ int num_contexts;
+ PrintDestination print_location;
memset(&grand_totals, 0, sizeof(grand_totals));
+ if (print_to_stderr)
+ print_location = PRINT_STATS_TO_STDERR;
+ else
+ print_location = PRINT_STATS_TO_LOGS;
+
+ /* num_contexts report number of contexts aggregated in the output */
MemoryContextStatsInternal(context, 1, max_level, max_children,
- &grand_totals, print_to_stderr);
+ &grand_totals, print_location, &num_contexts);
if (print_to_stderr)
fprintf(stderr,
@@ -870,13 +875,14 @@ MemoryContextStatsDetail(MemoryContext context,
* One recursion level for MemoryContextStats
*
* Print stats for this context if possible, but in any case accumulate counts
- * into *totals (if not NULL).
+ * into *totals (if not NULL). The callers should make sure that print_location
+ * is set to PRINT_STATS_TO_STDERR or PRINT_STATS_TO_LOGS or PRINT_STATS_NONE.
*/
-static void
+void
MemoryContextStatsInternal(MemoryContext context, int level,
int max_level, int max_children,
MemoryContextCounters *totals,
- bool print_to_stderr)
+ PrintDestination print_location, int *num_contexts)
{
MemoryContext child;
int ichild;
@@ -884,10 +890,39 @@ MemoryContextStatsInternal(MemoryContext context, int level,
Assert(MemoryContextIsValid(context));
/* Examine the context itself */
- context->methods->stats(context,
- MemoryContextStatsPrint,
- &level,
- totals, print_to_stderr);
+ switch (print_location)
+ {
+ case PRINT_STATS_TO_STDERR:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, true);
+ break;
+
+ case PRINT_STATS_TO_LOGS:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, false);
+ break;
+
+ case PRINT_STATS_NONE:
+
+ /*
+ * Do not print the statistics if print_location is
+ * PRINT_STATS_NONE, only compute totals. This is used in
+ * reporting of memory context statistics via a sql function. Last
+ * parameter is not relevant.
+ */
+ context->methods->stats(context,
+ NULL,
+ NULL,
+ totals, false);
+ break;
+ }
+
+ /* Increment the context count for each of the recursive call */
+ *num_contexts = *num_contexts + 1;
/*
* Examine children.
@@ -907,7 +942,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
MemoryContextStatsInternal(child, level + 1,
max_level, max_children,
totals,
- print_to_stderr);
+ print_location, num_contexts);
}
}
@@ -926,7 +961,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
child = MemoryContextTraverseNext(child, context);
}
- if (print_to_stderr)
+ /*
+ * Add the count of children contexts which are traversed in the
+ * non-recursive manner.
+ */
+ *num_contexts = *num_contexts + ichild;
+
+ if (print_location == PRINT_STATS_TO_STDERR)
{
for (int i = 0; i < level; i++)
fprintf(stderr, " ");
@@ -939,7 +980,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
local_totals.freechunks,
local_totals.totalspace - local_totals.freespace);
}
- else
+ else if (print_location == PRINT_STATS_TO_LOGS)
ereport(LOG_SERVER_ONLY,
(errhidestmt(true),
errhidecontext(true),
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 1fc19146f46..4bc4474b580 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8597,6 +8597,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool float8',
+ proallargtypes => '{int4,bool,float8,text,text,text,_int4,int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, timeout, name, ident, type, path, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..1e59a7f910f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index 08a72569ae5..638407adf39 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -221,6 +221,7 @@ typedef enum BuiltinTrancheIds
LWTRANCHE_XACT_SLRU,
LWTRANCHE_PARALLEL_VACUUM_DSA,
LWTRANCHE_AIO_URING_COMPLETION,
+ LWTRANCHE_MEMORY_CONTEXT_KEYS,
LWTRANCHE_FIRST_USER_DEFINED,
} BuiltinTrancheIds;
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 8abc26abce2..01835d56021 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,10 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "storage/condition_variable.h"
+#include "storage/lmgr.h"
+#include "utils/dsa.h"
+#include "lib/dshash.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,6 +51,23 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 64
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum size per context. Actual size may be lower as this assumes the worst
+ * case of deepest path and longest identifiers (name and ident, thus the
+ * multiplication by 2). The path depth is limited to 100 like for memory
+ * context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
/*
* Standard top-level memory contexts.
@@ -319,4 +339,74 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+/* Dynamic shared memory state for statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ int proc_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+ TimestampTz stats_timestamp;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * Used for storage of transient identifiers for pg_get_backend_memory_contexts
+ */
+typedef struct MemoryStatsContextId
+{
+ MemoryContext context;
+ int context_id;
+} MemoryStatsContextId;
+
+/*
+ * This is passed to MemoryContextStatsInternal to determine whether
+ * to print context statistics or not and where to print them logs or
+ * stderr.
+ */
+typedef enum PrintDestination
+{
+ PRINT_STATS_TO_STDERR = 0,
+ PRINT_STATS_TO_LOGS,
+ PRINT_STATS_NONE
+} PrintDestination;
+
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsInternal(MemoryContext context, int level,
+ int max_level, int max_children,
+ MemoryContextCounters *totals,
+ PrintDestination print_location,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 83228cfca29..ae17d028ed3 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -232,3 +232,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..d0917b6868e 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 83192038571..fedae342032 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1683,6 +1683,9 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-07-29 13:40 Rahila Syed <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-07-29 13:40 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Daniel Gustafsson <[email protected]>
Hi,
Please find attached an updated patch. It contains the following changes.
1. It needed a rebase as highlighted by cfbot
<https://cfbot.cputube.org/patch_5938.log;. The method for adding an
LWLock was updated in commit-2047ad068139f0b8c6da73d0b845ca9ba30fb33d, so
the patch has been adjusted to reflect this change.
2. Updated some comments to align with the latest patch design.
3. Eliminated an unnecessary assertion
Thank you,
Rahila Syed
On Fri, Jul 11, 2025 at 9:01 PM Rahila Syed <[email protected]> wrote:
> Hi,
>
> Please find attached the latest memory context statistics monitoring
> patch.
> It has been redesigned to address several issues highlighted in the thread
> [1]
> and [2].
>
> Here are some key highlights of the new design:
>
> - All DSA processing has been moved out of the CFI handler function. Now,
> all the dynamic shared memory
> needed to store the statistics is created and deleted in the client
> function. This change addresses concerns
> that DSA APIs are too high level to be safely called from interrupt
> handlers. There was also a concern that
> DSA API calls might not provide re-entrancy, which could cause issues if
> CFI is invoked from a DSA function
> in the future.
>
> - The static shared memory array has been replaced with a DSHASH table
> which now holds metadata such as
> pointers to actual statistics for each process.
>
> - dsm_registry.c APIs are used for creating and attaching to DSA and
> DSHASH table, which helps prevent code
> duplication.
>
> -To address the memory leak concern, we create an exclusive memory context
> under the NULL context, which
> does not fall under the TopMemoryContext tree, to handle all the memory
> allocations in ProcessGetMemoryContextInterrupt.
> This ensures the memory context created by the function does not affect
> its outcome.
> The memory context is reset at the end of the function, which helps
> prevent any memory leaks.
>
> - Changes made to the mcxt.c file have been relocated to mcxtfuncs.c,
> which now contains all the existing
> memory statistics-related functions along with the code for the proposed
> function.
>
> The overall flow of a request is as follows:
>
> 1. A client backend running the pg_get_process_memory_contexts function
> creates a DSA and allocates memory
> to store statistics, tracked by DSA pointer. This pointer is stored in a
> DSHASH entry for each client querying the
> statistics of any process.
> The client shares its DSHASH table key with the server process using a
> static shared array of keys indexed
> by the server's procNumber. It notifies the server process to publish
> statistics by using SendProcSignal.
>
> 2. When a PostgreSQL server process handles the request for memory
> statistics, the CFI function accesses the
> client hash key stored in its procNumber slot of the shared keys array.
> The server process then retrieves the
> DSHASH entry to obtain the DSA pointer allocated by the client, for
> storing the statistics.
> After storing the statistics, it notifies the client through its
> condition variable.
>
> 3. Although the DSA is created just once, the memory inside the DSA is
> allocated and released by the client
> process as soon as it finishes reading the statistics.
> If it fails to do so, it is deleted by the before_shmem_exit callback when
> the client exits. The client's entry in DSHASH
> table is also deleted when the client exits.
>
> 4. The DSA and DSHASH table are not created
> until pg_get_process_memory_context function is called.
> Once created, any client backend querying statistics and any PostgreSQL
> process publishing statistics will
> attach to the same area and table.
>
> Please let me know your thoughts.
>
> Thank you,
> Rahila Syed
>
> [1]. PostgreSQL: Re: pgsql: Add function to get memory context stats for
> processes
> <https://www.postgresql.org/message-id/CA%2BTgmoaey-kOP1k5FaUnQFd1fR0majVebWcL8ogfLbG_nt-Ytg%40mail.g...;
> [2]. PostgreSQL: Re: Prevent an error on attaching/creating a DSM/DSA
> from an interrupt handler.
> <https://www.postgresql.org/message-id/flat/8B873D49-E0E5-4F9F-B8D6-CA4836B825CD%40yesql.se#7026d2fe4...;
>
> On Wed, Apr 30, 2025 at 4:13 PM Daniel Gustafsson <[email protected]> wrote:
>
>> > On 30 Apr 2025, at 12:14, Peter Eisentraut <[email protected]>
>> wrote:
>> >
>> > On 29.04.25 15:13, Rahila Syed wrote:
>> >> Please find attached a patch with some comments and documentation
>> changes.
>> >> Additionaly, added a missing '\0' termination to "Remaining Totals"
>> string.
>> >> I think this became necessary after we replaced dsa_allocate0()
>> >> with dsa_allocate() is the latest version.
>> >
>> > > strncpy(nameptr, "Remaining Totals", namelen);
>> > > + nameptr[namelen] = '\0';
>> >
>> > Looks like a case for strlcpy()?
>>
>> True. I did go ahead with the strncpy and nul terminator assignment,
>> mostly
>> out of muscle memory, but I agree that this would be a good place for a
>> strlcpy() instead.
>>
>> --
>> Daniel Gustafsson
>>
>>
Attachments:
[application/octet-stream] v31-0001-Add-pg_get_process_memory_context-function.patch (59.6K, ../../CAH2L28vCCgye_+kJt22RAFzZfYbO7ytSrp-hR6-SenBcm_cN+w@mail.gmail.com/3-v31-0001-Add-pg_get_process_memory_context-function.patch)
download | inline diff:
From 9487a3d5d673bb0ec5521ad8bdb1e6cb603a51a6 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Mon, 30 Jun 2025 12:11:00 +0530
Subject: [PATCH] Add pg_get_process_memory_context function
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes the caller specifies
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned.
---
doc/src/sgml/func.sgml | 164 ++++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 831 +++++++++++++++++-
src/backend/utils/adt/pg_locale.c | 1 -
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mb/mbutils.c | 1 -
src/backend/utils/mmgr/mcxt.c | 71 +-
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 92 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 3 +
27 files changed, 1221 insertions(+), 41 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 74a16af04ad..2cde766072d 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -28719,6 +28719,137 @@ acl | {postgres=arwdDxtm/postgres,foo=r/postgres}
</para></entry>
</row>
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type>, <parameter>timeout</parameter> <type>float</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type>,
+ <parameter>stats_timestamp</parameter> <type>timestamptz</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ Busy processes can delay reporting memory context statistics,
+ <parameter>timeout</parameter> specifies the number of seconds
+ to wait for updated statistics. <parameter>timeout</parameter> can be
+ specified in fractions of a second.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
+
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
@@ -28858,6 +28989,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false, 0.5) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+path | {1}
+level | 1
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index f6eca09ee15..49740fadb6d 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -682,6 +682,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 9474095f271..5e7e8081c05 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -781,6 +781,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 8490148a47d..767fc6a0014 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index 0ae9bf906ec..f24f574e748 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..ac97a39447c 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -867,6 +867,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 777c9a8d555..5d14684f6b2 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 2fa045e6b0f..fe3d32e40b0 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -51,6 +51,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -150,6 +151,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -343,6 +345,7 @@ CreateOrAttachShmemStructs(void)
WaitEventCustomShmemInit();
InjectionPointShmemInit();
AioShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index a9bb540b55a..ce69e26d720 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..f194e6b3dcc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -50,6 +50,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index a297606cdd7..c8cac8811ee 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3534,6 +3534,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 0be307d2ca0..54f91a76a1b 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -161,6 +161,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -406,6 +407,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..32a03f12b1a 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,13 +15,38 @@
#include "postgres.h"
+#include "access/twophase.h"
+#include "catalog/pg_authid_d.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+#define CLIENT_KEY_SIZE 64
+
+static LWLock *client_keys_lock = NULL;
+static int *client_keys = NULL;
+static dshash_table *MemoryStatsDsHash = NULL;
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(MemoryStatsDSHashEntry *entry);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts, int max_levels);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
@@ -89,7 +114,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +168,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +183,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +229,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +256,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +264,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +345,748 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to MEMSTATS_WAIT_TIMEOUT, retry given that there is
+ * time left within the timeout specified by the user, before giving up and
+ * returning previously published statistics, if any. If no previous statistics
+ * exist, return NULL.
+ */
+#define MEMSTATS_WAIT_TIMEOUT 100
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ double timeout = PG_GETARG_FLOAT8(2);
+ PGPROC *proc;
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit(1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ ereport(WARNING,
+ errmsg("server process is processing previous request %d: %m", pid));
+ LWLockRelease(client_keys_lock);
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ ConditionVariableInit(&entry->memcxt_cv);
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+ entry->summary = summary;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m", pid));
+ PG_RETURN_NULL();
+ }
+
+ while (1)
+ {
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using the correct
+ * entry in the proc_id field.
+ *
+ * Make sure that the information belongs to pid we requested
+ * information for, Otherwise loop back and wait for the server
+ * process to finish publishing statistics.
+ *
+ */
+ if (entry->proc_id == pid)
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The process ending during memory context processing is not an
+ * error.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ memstats_dsa_cleanup(entry);
+ PG_RETURN_NULL();
+ }
+
+
+ /*
+ * Wait for the timeout as defined by the user. If no statistics are
+ * available within the allowed time then return NULL. The timer is
+ * defined in milliseconds since that's what the condition variable
+ * sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (timeout * 1000), WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(entry);
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ if (memcxt_info[i].name[0] != '\0')
+ {
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+ }
+ else
+ nulls[0] = true;
+
+ if (memcxt_info[i].ident[0] != '\0')
+ {
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ }
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum, path_length, INT4OID);
+ values[3] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[3] = true;
+
+ values[4] = Int32GetDatum(memcxt_info[i].levels);
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ memstats_dsa_cleanup(entry);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(MemoryStatsDSHashEntry *entry)
+{
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->proc_id = 0;
+}
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in
+ * per-process DSA pointers. These pointers are stored in a dshash table with
+ * key as requesting clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ bool summary = false;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ /* Retreive the client key fo publishing statistics */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ LWLockRelease(client_keys_lock);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ summary = entry->summary;
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry->proc_id = MyProcPid;
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1, 100);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &c,
+ HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsInternal(c, 1, 100, 100, &grand_totals,
+ PRINT_STATS_NONE, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts, 100);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ /* Notify waiting backends and return */
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1, 100);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name, "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id -
+ num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+
+ /* Notify waiting backends and return */
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+}
+
+/*
+ * Update timestamp and signal all the waiting client backends after copying
+ * all the statistics.
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+
+ /*
+ * Empty this processes slot, so other clients can request memory
+ * statistics
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts, int max_levels)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), max_levels);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+}
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 97c2ac1faf9..ab768a7a91f 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -45,7 +45,6 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
-#include "utils/relcache.h"
#include "utils/syscache.h"
#ifdef WIN32
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..fb3f2d21fa0 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -662,6 +662,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 886ecbad871..308016d7763 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -39,7 +39,6 @@
#include "mb/pg_wchar.h"
#include "utils/fmgrprotos.h"
#include "utils/memutils.h"
-#include "utils/relcache.h"
#include "varatt.h"
/*
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index ce01dce9861..a6eda19edce 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -23,6 +23,7 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "utils/hsearch.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/memutils_internal.h"
@@ -159,10 +160,6 @@ MemoryContext PortalContext = NULL;
static void MemoryContextDeleteOnly(MemoryContext context);
static void MemoryContextCallResetCallbacks(MemoryContext context);
-static void MemoryContextStatsInternal(MemoryContext context, int level,
- int max_level, int max_children,
- MemoryContextCounters *totals,
- bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
const char *stats_string,
bool print_to_stderr);
@@ -864,11 +861,19 @@ MemoryContextStatsDetail(MemoryContext context,
bool print_to_stderr)
{
MemoryContextCounters grand_totals;
+ int num_contexts;
+ PrintDestination print_location;
memset(&grand_totals, 0, sizeof(grand_totals));
+ if (print_to_stderr)
+ print_location = PRINT_STATS_TO_STDERR;
+ else
+ print_location = PRINT_STATS_TO_LOGS;
+
+ /* num_contexts report number of contexts aggregated in the output */
MemoryContextStatsInternal(context, 1, max_level, max_children,
- &grand_totals, print_to_stderr);
+ &grand_totals, print_location, &num_contexts);
if (print_to_stderr)
fprintf(stderr,
@@ -903,13 +908,14 @@ MemoryContextStatsDetail(MemoryContext context,
* One recursion level for MemoryContextStats
*
* Print stats for this context if possible, but in any case accumulate counts
- * into *totals (if not NULL).
+ * into *totals (if not NULL). The callers should make sure that print_location
+ * is set to PRINT_STATS_TO_STDERR or PRINT_STATS_TO_LOGS or PRINT_STATS_NONE.
*/
-static void
+void
MemoryContextStatsInternal(MemoryContext context, int level,
int max_level, int max_children,
MemoryContextCounters *totals,
- bool print_to_stderr)
+ PrintDestination print_location, int *num_contexts)
{
MemoryContext child;
int ichild;
@@ -917,10 +923,39 @@ MemoryContextStatsInternal(MemoryContext context, int level,
Assert(MemoryContextIsValid(context));
/* Examine the context itself */
- context->methods->stats(context,
- MemoryContextStatsPrint,
- &level,
- totals, print_to_stderr);
+ switch (print_location)
+ {
+ case PRINT_STATS_TO_STDERR:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, true);
+ break;
+
+ case PRINT_STATS_TO_LOGS:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, false);
+ break;
+
+ case PRINT_STATS_NONE:
+
+ /*
+ * Do not print the statistics if print_location is
+ * PRINT_STATS_NONE, only compute totals. This is used in
+ * reporting of memory context statistics via a sql function. Last
+ * parameter is not relevant.
+ */
+ context->methods->stats(context,
+ NULL,
+ NULL,
+ totals, false);
+ break;
+ }
+
+ /* Increment the context count for each of the recursive call */
+ *num_contexts = *num_contexts + 1;
/*
* Examine children.
@@ -940,7 +975,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
MemoryContextStatsInternal(child, level + 1,
max_level, max_children,
totals,
- print_to_stderr);
+ print_location, num_contexts);
}
}
@@ -959,7 +994,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
child = MemoryContextTraverseNext(child, context);
}
- if (print_to_stderr)
+ /*
+ * Add the count of children contexts which are traversed in the
+ * non-recursive manner.
+ */
+ *num_contexts = *num_contexts + ichild;
+
+ if (print_location == PRINT_STATS_TO_STDERR)
{
for (int i = 0; i < level; i++)
fprintf(stderr, " ");
@@ -972,7 +1013,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
local_totals.freechunks,
local_totals.totalspace - local_totals.freespace);
}
- else
+ else if (print_location == PRINT_STATS_TO_LOGS)
ereport(LOG_SERVER_ONLY,
(errhidestmt(true),
errhidecontext(true),
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 3ee8fed7e53..707c965ecf5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8597,6 +8597,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool float8',
+ proallargtypes => '{int4,bool,float8,text,text,text,_int4,int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, timeout, name, ident, type, path, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..1e59a7f910f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 208d2e3a8ed..72ace053e9d 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -135,3 +135,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 8abc26abce2..01835d56021 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,10 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "storage/condition_variable.h"
+#include "storage/lmgr.h"
+#include "utils/dsa.h"
+#include "lib/dshash.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,6 +51,23 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 64
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum size per context. Actual size may be lower as this assumes the worst
+ * case of deepest path and longest identifiers (name and ident, thus the
+ * multiplication by 2). The path depth is limited to 100 like for memory
+ * context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
/*
* Standard top-level memory contexts.
@@ -319,4 +339,74 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+/* Dynamic shared memory state for statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ int proc_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+ TimestampTz stats_timestamp;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * Used for storage of transient identifiers for pg_get_backend_memory_contexts
+ */
+typedef struct MemoryStatsContextId
+{
+ MemoryContext context;
+ int context_id;
+} MemoryStatsContextId;
+
+/*
+ * This is passed to MemoryContextStatsInternal to determine whether
+ * to print context statistics or not and where to print them logs or
+ * stderr.
+ */
+typedef enum PrintDestination
+{
+ PRINT_STATS_TO_STDERR = 0,
+ PRINT_STATS_TO_LOGS,
+ PRINT_STATS_NONE
+} PrintDestination;
+
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsInternal(MemoryContext context, int level,
+ int max_level, int max_children,
+ MemoryContextCounters *totals,
+ PrintDestination print_location,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 83228cfca29..ae17d028ed3 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -232,3 +232,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..d0917b6868e 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3daba26b237..e614ae25a8c 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1681,6 +1681,9 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-08-08 09:26 Rahila Syed <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-08-08 09:26 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Daniel Gustafsson <[email protected]>
Hi,
CFbot indicated that the patch requires a rebase, so I've attached an
updated version.
The documentation for this feature is now included in the new
func-admin.sgml file,
due to recent changes in the documentation of sql functions.
The following are results from a performance test:
pgbench is initialized as follows :
pgbench -i -s 100 postgres
Test1 -
pgbench -c 16 -j 16 postgres -T 100
TPS: 745.02 (average of 3 runs)
Test2-
pgbench -c 16 -j 16 postgres -T 100
while memory usage of any postgres process is monitored concurrently every
0.1 seconds,
using the following method:
SELECT * FROM pg_get_process_memory_contexts(
(SELECT pid FROM pg_stat_activity
ORDER BY random() LIMIT 1)
, false, 5);
TPS: 750.66 (average of 3 runs)
I have not observed any performance decline resulting from the concurrent
execution
of the memory monitoring function.
Thank you,
Rahila Syed
On Tue, Jul 29, 2025 at 7:10 PM Rahila Syed <[email protected]> wrote:
> Hi,
>
> Please find attached an updated patch. It contains the following changes.
>
> 1. It needed a rebase as highlighted by cfbot
> <https://cfbot.cputube.org/patch_5938.log;. The method for adding an
> LWLock was updated in commit-2047ad068139f0b8c6da73d0b845ca9ba30fb33d, so
> the patch has been adjusted to reflect this change.
> 2. Updated some comments to align with the latest patch design.
> 3. Eliminated an unnecessary assertion
>
> Thank you,
> Rahila Syed
>
> On Fri, Jul 11, 2025 at 9:01 PM Rahila Syed <[email protected]>
> wrote:
>
>> Hi,
>>
>> Please find attached the latest memory context statistics monitoring
>> patch.
>> It has been redesigned to address several issues highlighted in the
>> thread [1]
>> and [2].
>>
>> Here are some key highlights of the new design:
>>
>> - All DSA processing has been moved out of the CFI handler function. Now,
>> all the dynamic shared memory
>> needed to store the statistics is created and deleted in the client
>> function. This change addresses concerns
>> that DSA APIs are too high level to be safely called from interrupt
>> handlers. There was also a concern that
>> DSA API calls might not provide re-entrancy, which could cause issues if
>> CFI is invoked from a DSA function
>> in the future.
>>
>> - The static shared memory array has been replaced with a DSHASH table
>> which now holds metadata such as
>> pointers to actual statistics for each process.
>>
>> - dsm_registry.c APIs are used for creating and attaching to DSA and
>> DSHASH table, which helps prevent code
>> duplication.
>>
>> -To address the memory leak concern, we create an exclusive memory
>> context under the NULL context, which
>> does not fall under the TopMemoryContext tree, to handle all the memory
>> allocations in ProcessGetMemoryContextInterrupt.
>> This ensures the memory context created by the function does not affect
>> its outcome.
>> The memory context is reset at the end of the function, which helps
>> prevent any memory leaks.
>>
>> - Changes made to the mcxt.c file have been relocated to mcxtfuncs.c,
>> which now contains all the existing
>> memory statistics-related functions along with the code for the proposed
>> function.
>>
>> The overall flow of a request is as follows:
>>
>> 1. A client backend running the pg_get_process_memory_contexts function
>> creates a DSA and allocates memory
>> to store statistics, tracked by DSA pointer. This pointer is stored in a
>> DSHASH entry for each client querying the
>> statistics of any process.
>> The client shares its DSHASH table key with the server process using a
>> static shared array of keys indexed
>> by the server's procNumber. It notifies the server process to publish
>> statistics by using SendProcSignal.
>>
>> 2. When a PostgreSQL server process handles the request for memory
>> statistics, the CFI function accesses the
>> client hash key stored in its procNumber slot of the shared keys array.
>> The server process then retrieves the
>> DSHASH entry to obtain the DSA pointer allocated by the client, for
>> storing the statistics.
>> After storing the statistics, it notifies the client through its
>> condition variable.
>>
>> 3. Although the DSA is created just once, the memory inside the DSA is
>> allocated and released by the client
>> process as soon as it finishes reading the statistics.
>> If it fails to do so, it is deleted by the before_shmem_exit callback
>> when the client exits. The client's entry in DSHASH
>> table is also deleted when the client exits.
>>
>> 4. The DSA and DSHASH table are not created
>> until pg_get_process_memory_context function is called.
>> Once created, any client backend querying statistics and any PostgreSQL
>> process publishing statistics will
>> attach to the same area and table.
>>
>> Please let me know your thoughts.
>>
>> Thank you,
>> Rahila Syed
>>
>> [1]. PostgreSQL: Re: pgsql: Add function to get memory context stats for
>> processes
>> <https://www.postgresql.org/message-id/CA%2BTgmoaey-kOP1k5FaUnQFd1fR0majVebWcL8ogfLbG_nt-Ytg%40mail.g...;
>> [2]. PostgreSQL: Re: Prevent an error on attaching/creating a DSM/DSA
>> from an interrupt handler.
>> <https://www.postgresql.org/message-id/flat/8B873D49-E0E5-4F9F-B8D6-CA4836B825CD%40yesql.se#7026d2fe4...;
>>
>> On Wed, Apr 30, 2025 at 4:13 PM Daniel Gustafsson <[email protected]>
>> wrote:
>>
>>> > On 30 Apr 2025, at 12:14, Peter Eisentraut <[email protected]>
>>> wrote:
>>> >
>>> > On 29.04.25 15:13, Rahila Syed wrote:
>>> >> Please find attached a patch with some comments and documentation
>>> changes.
>>> >> Additionaly, added a missing '\0' termination to "Remaining Totals"
>>> string.
>>> >> I think this became necessary after we replaced dsa_allocate0()
>>> >> with dsa_allocate() is the latest version.
>>> >
>>> > > strncpy(nameptr, "Remaining Totals", namelen);
>>> > > + nameptr[namelen] = '\0';
>>> >
>>> > Looks like a case for strlcpy()?
>>>
>>> True. I did go ahead with the strncpy and nul terminator assignment,
>>> mostly
>>> out of muscle memory, but I agree that this would be a good place for a
>>> strlcpy() instead.
>>>
>>> --
>>> Daniel Gustafsson
>>>
>>>
Attachments:
[application/octet-stream] v32-0001-Add-pg_get_process_memory_context-function.patch (59.6K, ../../CAH2L28t=O1k+5wdcP88rgty3OLZisTU72WGH8Dp2MxJjwn7=fw@mail.gmail.com/3-v32-0001-Add-pg_get_process_memory_context-function.patch)
download | inline diff:
From 17873d9d45f4207a1c227b49801befbcdb49b93d Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Mon, 30 Jun 2025 12:11:00 +0530
Subject: [PATCH] Add pg_get_process_memory_context function
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes the caller specifies
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned.
---
doc/src/sgml/func/func-admin.sgml | 164 ++++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 831 +++++++++++++++++-
src/backend/utils/adt/pg_locale.c | 1 -
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mb/mbutils.c | 1 -
src/backend/utils/mmgr/mcxt.c | 71 +-
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 92 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 3 +
27 files changed, 1221 insertions(+), 41 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 446fdfe56f4..689e9231a7e 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,137 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type>, <parameter>timeout</parameter> <type>float</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type>,
+ <parameter>stats_timestamp</parameter> <type>timestamptz</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ Busy processes can delay reporting memory context statistics,
+ <parameter>timeout</parameter> specifies the number of seconds
+ to wait for updated statistics. <parameter>timeout</parameter> can be
+ specified in fractions of a second.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +433,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false, 0.5) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+path | {1}
+level | 1
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 1b3c5a55882..022586d6a55 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -682,6 +682,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ff96b36d710..3875f76564d 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e84e8663e96..5b3e08805bf 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index 0ae9bf906ec..f24f574e748 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..ac97a39447c 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -867,6 +867,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 777c9a8d555..5d14684f6b2 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 2fa045e6b0f..fe3d32e40b0 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -51,6 +51,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -150,6 +151,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -343,6 +345,7 @@ CreateOrAttachShmemStructs(void)
WaitEventCustomShmemInit();
InjectionPointShmemInit();
AioShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..f194e6b3dcc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -50,6 +50,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 0cecd464902..3933b6db607 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3534,6 +3534,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 0be307d2ca0..54f91a76a1b 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -161,6 +161,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -406,6 +407,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..32a03f12b1a 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,13 +15,38 @@
#include "postgres.h"
+#include "access/twophase.h"
+#include "catalog/pg_authid_d.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+#define CLIENT_KEY_SIZE 64
+
+static LWLock *client_keys_lock = NULL;
+static int *client_keys = NULL;
+static dshash_table *MemoryStatsDsHash = NULL;
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(MemoryStatsDSHashEntry *entry);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts, int max_levels);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
@@ -89,7 +114,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +168,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +183,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +229,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +256,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +264,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +345,748 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to MEMSTATS_WAIT_TIMEOUT, retry given that there is
+ * time left within the timeout specified by the user, before giving up and
+ * returning previously published statistics, if any. If no previous statistics
+ * exist, return NULL.
+ */
+#define MEMSTATS_WAIT_TIMEOUT 100
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ double timeout = PG_GETARG_FLOAT8(2);
+ PGPROC *proc;
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit(1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ ereport(WARNING,
+ errmsg("server process is processing previous request %d: %m", pid));
+ LWLockRelease(client_keys_lock);
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ ConditionVariableInit(&entry->memcxt_cv);
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+ entry->summary = summary;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m", pid));
+ PG_RETURN_NULL();
+ }
+
+ while (1)
+ {
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using the correct
+ * entry in the proc_id field.
+ *
+ * Make sure that the information belongs to pid we requested
+ * information for, Otherwise loop back and wait for the server
+ * process to finish publishing statistics.
+ *
+ */
+ if (entry->proc_id == pid)
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The process ending during memory context processing is not an
+ * error.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ memstats_dsa_cleanup(entry);
+ PG_RETURN_NULL();
+ }
+
+
+ /*
+ * Wait for the timeout as defined by the user. If no statistics are
+ * available within the allowed time then return NULL. The timer is
+ * defined in milliseconds since that's what the condition variable
+ * sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (timeout * 1000), WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(entry);
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ if (memcxt_info[i].name[0] != '\0')
+ {
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+ }
+ else
+ nulls[0] = true;
+
+ if (memcxt_info[i].ident[0] != '\0')
+ {
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ }
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum, path_length, INT4OID);
+ values[3] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[3] = true;
+
+ values[4] = Int32GetDatum(memcxt_info[i].levels);
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ memstats_dsa_cleanup(entry);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(MemoryStatsDSHashEntry *entry)
+{
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->proc_id = 0;
+}
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in
+ * per-process DSA pointers. These pointers are stored in a dshash table with
+ * key as requesting clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ bool summary = false;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ /* Retreive the client key fo publishing statistics */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ LWLockRelease(client_keys_lock);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ summary = entry->summary;
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry->proc_id = MyProcPid;
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1, 100);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &c,
+ HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsInternal(c, 1, 100, 100, &grand_totals,
+ PRINT_STATS_NONE, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts, 100);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ /* Notify waiting backends and return */
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1, 100);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name, "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id -
+ num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+
+ /* Notify waiting backends and return */
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+}
+
+/*
+ * Update timestamp and signal all the waiting client backends after copying
+ * all the statistics.
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+
+ /*
+ * Empty this processes slot, so other clients can request memory
+ * statistics
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts, int max_levels)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), max_levels);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+}
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 97c2ac1faf9..ab768a7a91f 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -45,7 +45,6 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
-#include "utils/relcache.h"
#include "utils/syscache.h"
#ifdef WIN32
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..fb3f2d21fa0 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -662,6 +662,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 886ecbad871..308016d7763 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -39,7 +39,6 @@
#include "mb/pg_wchar.h"
#include "utils/fmgrprotos.h"
#include "utils/memutils.h"
-#include "utils/relcache.h"
#include "varatt.h"
/*
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..3a5422c7273 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -40,6 +40,7 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "utils/hsearch.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/memutils_internal.h"
@@ -176,10 +177,6 @@ MemoryContext PortalContext = NULL;
static void MemoryContextDeleteOnly(MemoryContext context);
static void MemoryContextCallResetCallbacks(MemoryContext context);
-static void MemoryContextStatsInternal(MemoryContext context, int level,
- int max_level, int max_children,
- MemoryContextCounters *totals,
- bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
const char *stats_string,
bool print_to_stderr);
@@ -877,11 +874,19 @@ MemoryContextStatsDetail(MemoryContext context,
bool print_to_stderr)
{
MemoryContextCounters grand_totals;
+ int num_contexts;
+ PrintDestination print_location;
memset(&grand_totals, 0, sizeof(grand_totals));
+ if (print_to_stderr)
+ print_location = PRINT_STATS_TO_STDERR;
+ else
+ print_location = PRINT_STATS_TO_LOGS;
+
+ /* num_contexts report number of contexts aggregated in the output */
MemoryContextStatsInternal(context, 1, max_level, max_children,
- &grand_totals, print_to_stderr);
+ &grand_totals, print_location, &num_contexts);
if (print_to_stderr)
fprintf(stderr,
@@ -916,13 +921,14 @@ MemoryContextStatsDetail(MemoryContext context,
* One recursion level for MemoryContextStats
*
* Print stats for this context if possible, but in any case accumulate counts
- * into *totals (if not NULL).
+ * into *totals (if not NULL). The callers should make sure that print_location
+ * is set to PRINT_STATS_TO_STDERR or PRINT_STATS_TO_LOGS or PRINT_STATS_NONE.
*/
-static void
+void
MemoryContextStatsInternal(MemoryContext context, int level,
int max_level, int max_children,
MemoryContextCounters *totals,
- bool print_to_stderr)
+ PrintDestination print_location, int *num_contexts)
{
MemoryContext child;
int ichild;
@@ -930,10 +936,39 @@ MemoryContextStatsInternal(MemoryContext context, int level,
Assert(MemoryContextIsValid(context));
/* Examine the context itself */
- context->methods->stats(context,
- MemoryContextStatsPrint,
- &level,
- totals, print_to_stderr);
+ switch (print_location)
+ {
+ case PRINT_STATS_TO_STDERR:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, true);
+ break;
+
+ case PRINT_STATS_TO_LOGS:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, false);
+ break;
+
+ case PRINT_STATS_NONE:
+
+ /*
+ * Do not print the statistics if print_location is
+ * PRINT_STATS_NONE, only compute totals. This is used in
+ * reporting of memory context statistics via a sql function. Last
+ * parameter is not relevant.
+ */
+ context->methods->stats(context,
+ NULL,
+ NULL,
+ totals, false);
+ break;
+ }
+
+ /* Increment the context count for each of the recursive call */
+ *num_contexts = *num_contexts + 1;
/*
* Examine children.
@@ -953,7 +988,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
MemoryContextStatsInternal(child, level + 1,
max_level, max_children,
totals,
- print_to_stderr);
+ print_location, num_contexts);
}
}
@@ -972,7 +1007,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
child = MemoryContextTraverseNext(child, context);
}
- if (print_to_stderr)
+ /*
+ * Add the count of children contexts which are traversed in the
+ * non-recursive manner.
+ */
+ *num_contexts = *num_contexts + ichild;
+
+ if (print_location == PRINT_STATS_TO_STDERR)
{
for (int i = 0; i < level; i++)
fprintf(stderr, " ");
@@ -985,7 +1026,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
local_totals.freechunks,
local_totals.totalspace - local_totals.freespace);
}
- else
+ else if (print_location == PRINT_STATS_TO_LOGS)
ereport(LOG_SERVER_ONLY,
(errhidestmt(true),
errhidecontext(true),
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 118d6da1ace..3d6f42606fd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8597,6 +8597,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool float8',
+ proallargtypes => '{int4,bool,float8,text,text,text,_int4,int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, timeout, name, ident, type, path, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..1e59a7f910f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 208d2e3a8ed..72ace053e9d 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -135,3 +135,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 8abc26abce2..01835d56021 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,10 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "storage/condition_variable.h"
+#include "storage/lmgr.h"
+#include "utils/dsa.h"
+#include "lib/dshash.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,6 +51,23 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 64
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum size per context. Actual size may be lower as this assumes the worst
+ * case of deepest path and longest identifiers (name and ident, thus the
+ * multiplication by 2). The path depth is limited to 100 like for memory
+ * context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
/*
* Standard top-level memory contexts.
@@ -319,4 +339,74 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+/* Dynamic shared memory state for statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ int proc_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+ TimestampTz stats_timestamp;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * Used for storage of transient identifiers for pg_get_backend_memory_contexts
+ */
+typedef struct MemoryStatsContextId
+{
+ MemoryContext context;
+ int context_id;
+} MemoryStatsContextId;
+
+/*
+ * This is passed to MemoryContextStatsInternal to determine whether
+ * to print context statistics or not and where to print them logs or
+ * stderr.
+ */
+typedef enum PrintDestination
+{
+ PRINT_STATS_TO_STDERR = 0,
+ PRINT_STATS_TO_LOGS,
+ PRINT_STATS_NONE
+} PrintDestination;
+
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsInternal(MemoryContext context, int level,
+ int max_level, int max_children,
+ MemoryContextCounters *totals,
+ PrintDestination print_location,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 83228cfca29..ae17d028ed3 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -232,3 +232,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..d0917b6868e 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e6f2e93b2d6..d2399465df6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1680,6 +1680,9 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-08-12 00:34 torikoshia <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: torikoshia @ 2025-08-12 00:34 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: pgsql-hackers; Daniel Gustafsson <[email protected]>
On 2025-08-08 18:26, Rahila Syed wrote:
Hi, thanks for working on this again.
> Hi,
>
> CFbot indicated that the patch requires a rebase, so I've attached an
> updated version.
Here are some comments and questions for v32 patch:
> --- a/doc/src/sgml/func/func-admin.sgml
> +++ b/doc/src/sgml/func/func-admin.sgml
> @@ -251,6 +251,137 @@
> <literal>false</literal> is returned.
> </para></entry>
> </row>
> +
> + <row>
> + <entry role="func_table_entry"><para role="func_signature">
> + <indexterm>
> + <primary>pg_get_process_memory_contexts</primary>
This function is added at the end of Table "9.96. Server Signaling
Functions", but since pg_get_process_memory_contexts outputs essentially
the same information as pg_log_backend_memory_contexts, it might be
better to place them next to each other in the table.
> + <parameter>stats_timestamp</parameter> <type>timestamptz</type> )
> +typedef struct MemoryStatsDSHashEntry
> +{
> + char key[64];
> + ConditionVariable memcxt_cv;
> + int proc_id;
> + int total_stats;
> + bool summary;
> + dsa_pointer memstats_dsa_pointer;
> + TimestampTz stats_timestamp;
> +} MemoryStatsDSHashEntry;
stats_timestamp appears only in the two places below in the patch, but
it does not seem to be actually output.
Is this column unnecessary?
=# select * from pg_get_process_memory_contexts(pg_backend_pid(),
true, 10);
-[ RECORD 1 ]----+-----------------------------
name | TopMemoryContext
ident | [NULL]
type | AllocSet
path | {1}
level | 1
total_bytes | 222400
total_nblocks | 8
free_bytes | 4776
free_chunks | 8
used_bytes | 217624
num_agg_contexts | 1
Specifying 0 for timeout causes a crash:
=# select * from pg_get_process_memory_contexts(74526, true, 0);
(0 rows)
=# select 1;
WARNING: terminating connection because of crash of another server
process
DETAIL: The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and
repeat your command.
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Should 0 be handled safely and treated as “no timeout”, or rejected as
an error?
Similarly, specifying a negative value for timeout still works:
=# select * from pg_get_process_memory_contexts(30590, true, -10);
It might be better to reject negative values similar to
pg_terminate_backend().
> context_id_lookup =
> hash_create("pg_get_remote_backend_memory_contexts",
> + /* Retreive the client key fo publishing statistics */
fo -> for?
> + * If the publishing backend does not respond before the condition
> variable
> + * times out, which is set to MEMSTATS_WAIT_TIMEOUT, retry given that
> there is
> + * time left within the timeout specified by the user, before giving
> up and
> + * returning previously published statistics, if any. If no previous
> statistics
> + * exist, return NULL.
> + */
> +#define MEMSTATS_WAIT_TIMEOUT 100
MEMSTATS_WAIT_TIMEOUT is defined, but it doesn’t seem to be used.
--
Regards,
--
Atsushi Torikoshi
Seconded from NTT DATA Japan Corporation to SRA OSS K.K.
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-08-13 22:35 Rahila Syed <[email protected]>
parent: torikoshia <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-08-13 22:35 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi Torikoshia,
Thank you for reviewing the patch.
>
> This function is added at the end of Table "9.96. Server Signaling
> Functions", but since pg_get_process_memory_contexts outputs essentially
> the same information as pg_log_backend_memory_contexts, it might be
> better to place them next to each other in the table.
>
>
The idea was to place the new addition at the end of the table instead of
in the middle.
I’m fine with putting them together, though. I’ll do that in the next
version unless there’s a
reason not to.
>
> > + <parameter>stats_timestamp</parameter> <type>timestamptz</type> )
>
> > +typedef struct MemoryStatsDSHashEntry
> > +{
> > + char key[64];
> > + ConditionVariable memcxt_cv;
> > + int proc_id;
> > + int total_stats;
> > + bool summary;
> > + dsa_pointer memstats_dsa_pointer;
> > + TimestampTz stats_timestamp;
> > +} MemoryStatsDSHashEntry;
>
> stats_timestamp appears only in the two places below in the patch, but
> it does not seem to be actually output.
> Is this column unnecessary?
>
>
Thank you for pointing this out. This is removed in the attached patch, as
it was a
remnant from the previous design. As old statistics are discarded in the
new design,
a timestamp field is not needed anymore.
>
> Specifying 0 for timeout causes a crash:
> Should 0 be handled safely and treated as “no timeout”, or rejected as
> an error?
>
Good catch.
The crash has been resolved in the attached patch. It was caused by a
missing
ConditionVariableCancelSleep() call when exiting without statistics due to
a timeout value of 0.
A 0 timeout means that statistics should only be retrieved if they are
immediately available,
without waiting. We could exit with a warning/error saying "too low
timeout", but I think it's worthwhile
to try fetching the statistics if possible.
Similarly, specifying a negative value for timeout still works:
>
> =# select * from pg_get_process_memory_contexts(30590, true, -10);
>
> It might be better to reject negative values similar to
> pg_terminate_backend().
>
>
>
Fixed as suggested by you in the attached patch.
Currently, negative values are interpreted as an indefinite wait for
statistics.
This could cause the client to hang if the server process exits without
providing statistics.
To avoid this, it would be better to exit after displaying a warning when
the user specifies
negative timeouts.
>
> > + /* Retreive the client key fo publishing statistics */
>
> fo -> for?
>
Fixed.
> + */
> > +#define MEMSTATS_WAIT_TIMEOUT 100
>
> MEMSTATS_WAIT_TIMEOUT is defined, but it doesn’t seem to be used.
>
>
This is removed now as it was a leftover from the previous design.
The attached patch also fixes an assertion failure I observed when a client
times out
before the last requested process can publish its statistics. A client
frees the memory
reserved for storing the statistics when it exits the function after
timeout. Since a
server process was notified, it might attempt to read the same client entry
and access the dsa
memory reserved for statistics resulting in the assertion
failure. I resolved this by including a check for this scenario and then
exiting the handler
function accordingly.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v33-0001-Add-pg_get_process_memory_context-function.patch (61.1K, ../../CAH2L28sF02nZj5vEL49EsTx+TfYpB3dFgShgn0=W1V-uFZjejw@mail.gmail.com/3-v33-0001-Add-pg_get_process_memory_context-function.patch)
download | inline diff:
From f37e942fa7732795dc37ef5ef3759622f1b71eef Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Mon, 30 Jun 2025 12:11:00 +0530
Subject: [PATCH] Add pg_get_process_memory_context function
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes the caller specifies
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned.
---
doc/src/sgml/func/func-admin.sgml | 164 ++++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 887 +++++++++++++++++-
src/backend/utils/adt/pg_locale.c | 1 -
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mb/mbutils.c | 1 -
src/backend/utils/mmgr/mcxt.c | 71 +-
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 92 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 3 +
27 files changed, 1277 insertions(+), 41 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 446fdfe56f4..689e9231a7e 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,137 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type>, <parameter>timeout</parameter> <type>float</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type>,
+ <parameter>stats_timestamp</parameter> <type>timestamptz</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ Busy processes can delay reporting memory context statistics,
+ <parameter>timeout</parameter> specifies the number of seconds
+ to wait for updated statistics. <parameter>timeout</parameter> can be
+ specified in fractions of a second.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +433,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false, 0.5) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+path | {1}
+level | 1
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 1b3c5a55882..022586d6a55 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -682,6 +682,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ff96b36d710..3875f76564d 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e84e8663e96..5b3e08805bf 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index 0ae9bf906ec..f24f574e748 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..ac97a39447c 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -867,6 +867,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 777c9a8d555..5d14684f6b2 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 2fa045e6b0f..fe3d32e40b0 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -51,6 +51,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -150,6 +151,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -343,6 +345,7 @@ CreateOrAttachShmemStructs(void)
WaitEventCustomShmemInit();
InjectionPointShmemInit();
AioShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..f194e6b3dcc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -50,6 +50,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 0cecd464902..3933b6db607 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3534,6 +3534,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 0be307d2ca0..54f91a76a1b 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -161,6 +161,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -406,6 +407,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..ce507be3e85 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,13 +15,38 @@
#include "postgres.h"
+#include "access/twophase.h"
+#include "catalog/pg_authid_d.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+#define CLIENT_KEY_SIZE 64
+
+static LWLock *client_keys_lock = NULL;
+static int *client_keys = NULL;
+static dshash_table *MemoryStatsDsHash = NULL;
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(MemoryStatsDSHashEntry *entry);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts, int max_levels);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
@@ -89,7 +114,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +168,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +183,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +229,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +256,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +264,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +345,804 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to timeout value specified by the user, give up and
+ * return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ double timeout = PG_GETARG_FLOAT8(2);
+ PGPROC *proc;
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+
+ if (timeout < 0)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ (errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("\"timeout\" must not be negative")));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit(1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request", pid));
+ LWLockRelease(client_keys_lock);
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from.
+ * If this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since the
+ * previous server was notified, it might attempt to read the same client entry
+ * and respond incorrectly with its statistics. By storing the server ID in the
+ * client entry, we prevent any previously signalled server process from writing
+ * its statistics in the space meant for the newly requested process.
+ */
+ entry->server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ PG_TRY();
+ {
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m", pid));
+ memstats_dsa_cleanup(entry);
+ PG_RETURN_NULL();
+ }
+
+ while (1)
+ {
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ */
+ if (entry->stats_written)
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Recheck the state of the backend before sleeping on the
+ * condition variable to ensure the process is still alive. Only
+ * check the relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The process ending during memory context processing is not an
+ * error.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ memstats_dsa_cleanup(entry);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+
+
+ /*
+ * Wait for the timeout as defined by the user. If no statistics
+ * are available within the allowed time then return NULL. The
+ * timer is defined in milliseconds since that's what the
+ * condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (timeout * 1000), WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(entry);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ if (memcxt_info[i].name[0] != '\0')
+ {
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+ }
+ else
+ nulls[0] = true;
+
+ if (memcxt_info[i].ident[0] != '\0')
+ {
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ }
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum, path_length, INT4OID);
+ values[3] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[3] = true;
+
+ values[4] = Int32GetDatum(memcxt_info[i].levels);
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ memstats_dsa_cleanup(entry);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ ConditionVariableCancelSleep();
+
+ }
+ PG_CATCH();
+ {
+ memstats_dsa_cleanup(entry);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ ConditionVariableCancelSleep();
+ }
+ PG_END_TRY();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(MemoryStatsDSHashEntry *entry)
+{
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->server_id = 0;
+}
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in
+ * per-process DSA pointers. These pointers are stored in a dshash table with
+ * key as requesting clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ bool summary = false;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ /* Retreive the client key for publishing statistics */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ LWLockRelease(client_keys_lock);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /* Entry has been deleted due to client process exit */
+ if (!found)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /* The client has timed out waiting for us to write statistics */
+ if (entry->server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ summary = entry->summary;
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1, 100);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &c,
+ HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsInternal(c, 1, 100, 100, &grand_totals,
+ PRINT_STATS_NONE, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts, 100);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ /* Notify waiting backends and return */
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1, 100);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name, "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id -
+ num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+}
+
+/*
+ * Update timestamp and signal all the waiting client backends after copying
+ * all the statistics.
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Empty this processes slot, so other clients can request memory
+ * statistics
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts, int max_levels)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), max_levels);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 97c2ac1faf9..ab768a7a91f 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -45,7 +45,6 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
-#include "utils/relcache.h"
#include "utils/syscache.h"
#ifdef WIN32
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..fb3f2d21fa0 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -662,6 +662,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 886ecbad871..308016d7763 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -39,7 +39,6 @@
#include "mb/pg_wchar.h"
#include "utils/fmgrprotos.h"
#include "utils/memutils.h"
-#include "utils/relcache.h"
#include "varatt.h"
/*
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..3a5422c7273 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -40,6 +40,7 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "utils/hsearch.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/memutils_internal.h"
@@ -176,10 +177,6 @@ MemoryContext PortalContext = NULL;
static void MemoryContextDeleteOnly(MemoryContext context);
static void MemoryContextCallResetCallbacks(MemoryContext context);
-static void MemoryContextStatsInternal(MemoryContext context, int level,
- int max_level, int max_children,
- MemoryContextCounters *totals,
- bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
const char *stats_string,
bool print_to_stderr);
@@ -877,11 +874,19 @@ MemoryContextStatsDetail(MemoryContext context,
bool print_to_stderr)
{
MemoryContextCounters grand_totals;
+ int num_contexts;
+ PrintDestination print_location;
memset(&grand_totals, 0, sizeof(grand_totals));
+ if (print_to_stderr)
+ print_location = PRINT_STATS_TO_STDERR;
+ else
+ print_location = PRINT_STATS_TO_LOGS;
+
+ /* num_contexts report number of contexts aggregated in the output */
MemoryContextStatsInternal(context, 1, max_level, max_children,
- &grand_totals, print_to_stderr);
+ &grand_totals, print_location, &num_contexts);
if (print_to_stderr)
fprintf(stderr,
@@ -916,13 +921,14 @@ MemoryContextStatsDetail(MemoryContext context,
* One recursion level for MemoryContextStats
*
* Print stats for this context if possible, but in any case accumulate counts
- * into *totals (if not NULL).
+ * into *totals (if not NULL). The callers should make sure that print_location
+ * is set to PRINT_STATS_TO_STDERR or PRINT_STATS_TO_LOGS or PRINT_STATS_NONE.
*/
-static void
+void
MemoryContextStatsInternal(MemoryContext context, int level,
int max_level, int max_children,
MemoryContextCounters *totals,
- bool print_to_stderr)
+ PrintDestination print_location, int *num_contexts)
{
MemoryContext child;
int ichild;
@@ -930,10 +936,39 @@ MemoryContextStatsInternal(MemoryContext context, int level,
Assert(MemoryContextIsValid(context));
/* Examine the context itself */
- context->methods->stats(context,
- MemoryContextStatsPrint,
- &level,
- totals, print_to_stderr);
+ switch (print_location)
+ {
+ case PRINT_STATS_TO_STDERR:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, true);
+ break;
+
+ case PRINT_STATS_TO_LOGS:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, false);
+ break;
+
+ case PRINT_STATS_NONE:
+
+ /*
+ * Do not print the statistics if print_location is
+ * PRINT_STATS_NONE, only compute totals. This is used in
+ * reporting of memory context statistics via a sql function. Last
+ * parameter is not relevant.
+ */
+ context->methods->stats(context,
+ NULL,
+ NULL,
+ totals, false);
+ break;
+ }
+
+ /* Increment the context count for each of the recursive call */
+ *num_contexts = *num_contexts + 1;
/*
* Examine children.
@@ -953,7 +988,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
MemoryContextStatsInternal(child, level + 1,
max_level, max_children,
totals,
- print_to_stderr);
+ print_location, num_contexts);
}
}
@@ -972,7 +1007,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
child = MemoryContextTraverseNext(child, context);
}
- if (print_to_stderr)
+ /*
+ * Add the count of children contexts which are traversed in the
+ * non-recursive manner.
+ */
+ *num_contexts = *num_contexts + ichild;
+
+ if (print_location == PRINT_STATS_TO_STDERR)
{
for (int i = 0; i < level; i++)
fprintf(stderr, " ");
@@ -985,7 +1026,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
local_totals.freechunks,
local_totals.totalspace - local_totals.freespace);
}
- else
+ else if (print_location == PRINT_STATS_TO_LOGS)
ereport(LOG_SERVER_ONLY,
(errhidestmt(true),
errhidecontext(true),
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 118d6da1ace..3d6f42606fd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8597,6 +8597,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool float8',
+ proallargtypes => '{int4,bool,float8,text,text,text,_int4,int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, timeout, name, ident, type, path, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..1e59a7f910f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 208d2e3a8ed..72ace053e9d 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -135,3 +135,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 8abc26abce2..799cfe38cab 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,10 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "storage/condition_variable.h"
+#include "storage/lmgr.h"
+#include "utils/dsa.h"
+#include "lib/dshash.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,6 +51,23 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 64
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum size per context. Actual size may be lower as this assumes the worst
+ * case of deepest path and longest identifiers (name and ident, thus the
+ * multiplication by 2). The path depth is limited to 100 like for memory
+ * context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
/*
* Standard top-level memory contexts.
@@ -319,4 +339,74 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+/* Dynamic shared memory state for statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * Used for storage of transient identifiers for pg_get_backend_memory_contexts
+ */
+typedef struct MemoryStatsContextId
+{
+ MemoryContext context;
+ int context_id;
+} MemoryStatsContextId;
+
+/*
+ * This is passed to MemoryContextStatsInternal to determine whether
+ * to print context statistics or not and where to print them logs or
+ * stderr.
+ */
+typedef enum PrintDestination
+{
+ PRINT_STATS_TO_STDERR = 0,
+ PRINT_STATS_TO_LOGS,
+ PRINT_STATS_NONE
+} PrintDestination;
+
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsInternal(MemoryContext context, int level,
+ int max_level, int max_children,
+ MemoryContextCounters *totals,
+ PrintDestination print_location,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 83228cfca29..ae17d028ed3 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -232,3 +232,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..d0917b6868e 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e6f2e93b2d6..d2399465df6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1680,6 +1680,9 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-08-18 13:32 torikoshia <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: torikoshia @ 2025-08-18 13:32 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: pgsql-hackers; Daniel Gustafsson <[email protected]>
On 2025-08-14 07:35, Rahila Syed wrote:
> Hi Torikoshia,
>
> Thank you for reviewing the patch.
>
>> This function is added at the end of Table "9.96. Server Signaling
>> Functions", but since pg_get_process_memory_contexts outputs
>> essentially
>> the same information as pg_log_backend_memory_contexts, it might be
>> better to place them next to each other in the table.
>
> The idea was to place the new addition at the end of the table instead
> of in the middle.
> I’m fine with putting them together, though. I’ll do that in the
> next version unless there’s a
> reason not to.
>
>>> + <parameter>stats_timestamp</parameter>
>> <type>timestamptz</type> )
>>
>>> +typedef struct MemoryStatsDSHashEntry
>>> +{
>>> + char key[64];
>>> + ConditionVariable memcxt_cv;
>>> + int proc_id;
>>> + int total_stats;
>>> + bool summary;
>>> + dsa_pointer memstats_dsa_pointer;
>>> + TimestampTz stats_timestamp;
>>> +} MemoryStatsDSHashEntry;
>>
>> stats_timestamp appears only in the two places below in the patch,
>> but
>> it does not seem to be actually output.
>> Is this column unnecessary?
>
> Thank you for pointing this out. This is removed in the attached
> patch, as it was a
> remnant from the previous design. As old statistics are discarded in
> the new design,
> a timestamp field is not needed anymore.
>
>> Specifying 0 for timeout causes a crash:
>> Should 0 be handled safely and treated as “no timeout”, or
>> rejected as
>> an error?
>
> Good catch.
> The crash has been resolved in the attached patch. It was caused by a
> missing
> ConditionVariableCancelSleep() call when exiting without statistics
> due to a timeout value of 0.
> A 0 timeout means that statistics should only be retrieved if they are
> immediately available,
> without waiting. We could exit with a warning/error saying "too low
> timeout", but I think it's worthwhile
> to try fetching the statistics if possible.
>
>> Similarly, specifying a negative value for timeout still works:
>>
>> =# select * from pg_get_process_memory_contexts(30590, true,
>> -10);
>>
>> It might be better to reject negative values similar to
>> pg_terminate_backend().
>
> Fixed as suggested by you in the attached patch.
> Currently, negative values are interpreted as an indefinite wait for
> statistics.
> This could cause the client to hang if the server process exits
> without providing statistics.
> To avoid this, it would be better to exit after displaying a warning
> when the user specifies
> negative timeouts.
>
>>> + /* Retreive the client key fo publishing statistics */
>>
>> fo -> for?
>
> Fixed.
>
>>> + */
>>> +#define MEMSTATS_WAIT_TIMEOUT 100
>>
>> MEMSTATS_WAIT_TIMEOUT is defined, but it doesn’t seem to be used.
>
> This is removed now as it was a leftover from the previous design.
>
> The attached patch also fixes an assertion failure I observed when a
> client times out
> before the last requested process can publish its statistics. A client
> frees the memory
> reserved for storing the statistics when it exits the function after
> timeout. Since a
> server process was notified, it might attempt to read the same client
> entry and access the dsa
> memory reserved for statistics resulting in the assertion
> failure. I resolved this by including a check for this scenario and
> then exiting the handler
> function accordingly.
Thanks for updating the patch!
However, when I ran pg_get_process_memory_contexts() with summary =
true, it took a while and returned nothing:
=# select pg_get_process_memory_contexts(pg_backend_pid(), true, 1)
from pg_stat_activity ;
pg_get_process_memory_contexts
--------------------------------
(0 rows)
Time: 6026.291 ms (00:06.026)
Since v32 patch quickly returned the memory contexts as expected with
the same parameter specified, there seems to be some degradation. Could
you check it?
--
Regards,
--
Atsushi Torikoshi
Seconded from NTT DATA Japan Corporation to SRA OSS K.K.
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-08-19 21:42 Rahila Syed <[email protected]>
parent: torikoshia <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-08-19 21:42 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi,
=# select pg_get_process_memory_contexts(pg_backend_pid(), true, 1)
> from pg_stat_activity ;
>
> pg_get_process_memory_contexts
> --------------------------------
> (0 rows)
>
> Time: 6026.291 ms (00:06.026)
>
> Since v32 patch quickly returned the memory contexts as expected with
> the same parameter specified, there seems to be some degradation. Could
> you check it?
>
Thank you for reporting this failure. This issue was a regression caused by
the absence of a
ConditionVariableSignal() call in the summary = true code path,
which happened due to recent code refactoring.
PFA the fix.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v34-0001-Add-pg_get_process_memory_context-function.patch (61.2K, ../../CAH2L28uL=Vm9E4Uf=tkg2Ki7_uDuAXORHVyCA7K_L=PAQ2p4sA@mail.gmail.com/3-v34-0001-Add-pg_get_process_memory_context-function.patch)
download | inline diff:
From ae939692b3bb507484136a3f32c239b6976ce0c2 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Mon, 30 Jun 2025 12:11:00 +0530
Subject: [PATCH] Add pg_get_process_memory_context function
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes the caller specifies
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned.
---
doc/src/sgml/func/func-admin.sgml | 164 ++++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 889 +++++++++++++++++-
src/backend/utils/adt/pg_locale.c | 1 -
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mb/mbutils.c | 1 -
src/backend/utils/mmgr/mcxt.c | 71 +-
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 92 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 3 +
27 files changed, 1279 insertions(+), 41 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 6347fe60b0c..43131c60882 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,137 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type>, <parameter>timeout</parameter> <type>float</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type>,
+ <parameter>stats_timestamp</parameter> <type>timestamptz</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ Busy processes can delay reporting memory context statistics,
+ <parameter>timeout</parameter> specifies the number of seconds
+ to wait for updated statistics. <parameter>timeout</parameter> can be
+ specified in fractions of a second.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +433,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false, 0.5) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+path | {1}
+level | 1
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 1b3c5a55882..022586d6a55 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -682,6 +682,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ff96b36d710..3875f76564d 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e84e8663e96..5b3e08805bf 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index 0ae9bf906ec..f24f574e748 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..ac97a39447c 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -867,6 +867,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 777c9a8d555..5d14684f6b2 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 2fa045e6b0f..fe3d32e40b0 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -51,6 +51,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -150,6 +151,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -343,6 +345,7 @@ CreateOrAttachShmemStructs(void)
WaitEventCustomShmemInit();
InjectionPointShmemInit();
AioShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index e9ef0fbfe32..f194e6b3dcc 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -50,6 +50,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 0cecd464902..3933b6db607 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3534,6 +3534,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 0be307d2ca0..54f91a76a1b 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -161,6 +161,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -406,6 +407,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..e465e156f77 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,13 +15,38 @@
#include "postgres.h"
+#include "access/twophase.h"
+#include "catalog/pg_authid_d.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+#define CLIENT_KEY_SIZE 64
+
+static LWLock *client_keys_lock = NULL;
+static int *client_keys = NULL;
+static dshash_table *MemoryStatsDsHash = NULL;
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(MemoryStatsDSHashEntry *entry);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts, int max_levels);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
@@ -89,7 +114,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +168,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +183,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +229,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +256,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +264,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +345,806 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to timeout value specified by the user, give up and
+ * return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ double timeout = PG_GETARG_FLOAT8(2);
+ PGPROC *proc;
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+
+ if (timeout < 0)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ (errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("\"timeout\" must not be negative")));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit(1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request", pid));
+ LWLockRelease(client_keys_lock);
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from.
+ * If this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since the
+ * previous server was notified, it might attempt to read the same client entry
+ * and respond incorrectly with its statistics. By storing the server ID in the
+ * client entry, we prevent any previously signalled server process from writing
+ * its statistics in the space meant for the newly requested process.
+ */
+ entry->server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ PG_TRY();
+ {
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m", pid));
+ memstats_dsa_cleanup(entry);
+ PG_RETURN_NULL();
+ }
+
+ while (1)
+ {
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ */
+ if (entry->stats_written)
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Recheck the state of the backend before sleeping on the
+ * condition variable to ensure the process is still alive. Only
+ * check the relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The process ending during memory context processing is not an
+ * error.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ memstats_dsa_cleanup(entry);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+
+
+ /*
+ * Wait for the timeout as defined by the user. If no statistics
+ * are available within the allowed time then return NULL. The
+ * timer is defined in milliseconds since that's what the
+ * condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (timeout * 1000), WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(entry);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ if (memcxt_info[i].name[0] != '\0')
+ {
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+ }
+ else
+ nulls[0] = true;
+
+ if (memcxt_info[i].ident[0] != '\0')
+ {
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ }
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum, path_length, INT4OID);
+ values[3] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[3] = true;
+
+ values[4] = Int32GetDatum(memcxt_info[i].levels);
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ memstats_dsa_cleanup(entry);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ ConditionVariableCancelSleep();
+
+ }
+ PG_CATCH();
+ {
+ memstats_dsa_cleanup(entry);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ ConditionVariableCancelSleep();
+ }
+ PG_END_TRY();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(MemoryStatsDSHashEntry *entry)
+{
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->server_id = 0;
+}
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in
+ * per-process DSA pointers. These pointers are stored in a dshash table with
+ * key as requesting clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ bool summary = false;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ /* Retreive the client key for publishing statistics */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ LWLockRelease(client_keys_lock);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /* Entry has been deleted due to client process exit */
+ if (!found)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /* The client has timed out waiting for us to write statistics */
+ if (entry->server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ summary = entry->summary;
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1, 100);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &c,
+ HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsInternal(c, 1, 100, 100, &grand_totals,
+ PRINT_STATS_NONE, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts, 100);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1, 100);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name, "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id -
+ num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+}
+
+/*
+ * Update timestamp and signal all the waiting client backends after copying
+ * all the statistics.
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Empty this processes slot, so other clients can request memory
+ * statistics
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts, int max_levels)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), max_levels);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 97c2ac1faf9..ab768a7a91f 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -45,7 +45,6 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
-#include "utils/relcache.h"
#include "utils/syscache.h"
#ifdef WIN32
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..fb3f2d21fa0 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -662,6 +662,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 886ecbad871..308016d7763 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -39,7 +39,6 @@
#include "mb/pg_wchar.h"
#include "utils/fmgrprotos.h"
#include "utils/memutils.h"
-#include "utils/relcache.h"
#include "varatt.h"
/*
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..3a5422c7273 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -40,6 +40,7 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "utils/hsearch.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/memutils_internal.h"
@@ -176,10 +177,6 @@ MemoryContext PortalContext = NULL;
static void MemoryContextDeleteOnly(MemoryContext context);
static void MemoryContextCallResetCallbacks(MemoryContext context);
-static void MemoryContextStatsInternal(MemoryContext context, int level,
- int max_level, int max_children,
- MemoryContextCounters *totals,
- bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
const char *stats_string,
bool print_to_stderr);
@@ -877,11 +874,19 @@ MemoryContextStatsDetail(MemoryContext context,
bool print_to_stderr)
{
MemoryContextCounters grand_totals;
+ int num_contexts;
+ PrintDestination print_location;
memset(&grand_totals, 0, sizeof(grand_totals));
+ if (print_to_stderr)
+ print_location = PRINT_STATS_TO_STDERR;
+ else
+ print_location = PRINT_STATS_TO_LOGS;
+
+ /* num_contexts report number of contexts aggregated in the output */
MemoryContextStatsInternal(context, 1, max_level, max_children,
- &grand_totals, print_to_stderr);
+ &grand_totals, print_location, &num_contexts);
if (print_to_stderr)
fprintf(stderr,
@@ -916,13 +921,14 @@ MemoryContextStatsDetail(MemoryContext context,
* One recursion level for MemoryContextStats
*
* Print stats for this context if possible, but in any case accumulate counts
- * into *totals (if not NULL).
+ * into *totals (if not NULL). The callers should make sure that print_location
+ * is set to PRINT_STATS_TO_STDERR or PRINT_STATS_TO_LOGS or PRINT_STATS_NONE.
*/
-static void
+void
MemoryContextStatsInternal(MemoryContext context, int level,
int max_level, int max_children,
MemoryContextCounters *totals,
- bool print_to_stderr)
+ PrintDestination print_location, int *num_contexts)
{
MemoryContext child;
int ichild;
@@ -930,10 +936,39 @@ MemoryContextStatsInternal(MemoryContext context, int level,
Assert(MemoryContextIsValid(context));
/* Examine the context itself */
- context->methods->stats(context,
- MemoryContextStatsPrint,
- &level,
- totals, print_to_stderr);
+ switch (print_location)
+ {
+ case PRINT_STATS_TO_STDERR:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, true);
+ break;
+
+ case PRINT_STATS_TO_LOGS:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, false);
+ break;
+
+ case PRINT_STATS_NONE:
+
+ /*
+ * Do not print the statistics if print_location is
+ * PRINT_STATS_NONE, only compute totals. This is used in
+ * reporting of memory context statistics via a sql function. Last
+ * parameter is not relevant.
+ */
+ context->methods->stats(context,
+ NULL,
+ NULL,
+ totals, false);
+ break;
+ }
+
+ /* Increment the context count for each of the recursive call */
+ *num_contexts = *num_contexts + 1;
/*
* Examine children.
@@ -953,7 +988,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
MemoryContextStatsInternal(child, level + 1,
max_level, max_children,
totals,
- print_to_stderr);
+ print_location, num_contexts);
}
}
@@ -972,7 +1007,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
child = MemoryContextTraverseNext(child, context);
}
- if (print_to_stderr)
+ /*
+ * Add the count of children contexts which are traversed in the
+ * non-recursive manner.
+ */
+ *num_contexts = *num_contexts + ichild;
+
+ if (print_location == PRINT_STATS_TO_STDERR)
{
for (int i = 0; i < level; i++)
fprintf(stderr, " ");
@@ -985,7 +1026,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
local_totals.freechunks,
local_totals.totalspace - local_totals.freespace);
}
- else
+ else if (print_location == PRINT_STATS_TO_LOGS)
ereport(LOG_SERVER_ONLY,
(errhidestmt(true),
errhidecontext(true),
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 118d6da1ace..3d6f42606fd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8597,6 +8597,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool float8',
+ proallargtypes => '{int4,bool,float8,text,text,text,_int4,int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, timeout, name, ident, type, path, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..1e59a7f910f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 208d2e3a8ed..72ace053e9d 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -135,3 +135,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 8abc26abce2..799cfe38cab 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,10 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "storage/condition_variable.h"
+#include "storage/lmgr.h"
+#include "utils/dsa.h"
+#include "lib/dshash.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,6 +51,23 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 64
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum size per context. Actual size may be lower as this assumes the worst
+ * case of deepest path and longest identifiers (name and ident, thus the
+ * multiplication by 2). The path depth is limited to 100 like for memory
+ * context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
/*
* Standard top-level memory contexts.
@@ -319,4 +339,74 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+/* Dynamic shared memory state for statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * Used for storage of transient identifiers for pg_get_backend_memory_contexts
+ */
+typedef struct MemoryStatsContextId
+{
+ MemoryContext context;
+ int context_id;
+} MemoryStatsContextId;
+
+/*
+ * This is passed to MemoryContextStatsInternal to determine whether
+ * to print context statistics or not and where to print them logs or
+ * stderr.
+ */
+typedef enum PrintDestination
+{
+ PRINT_STATS_TO_STDERR = 0,
+ PRINT_STATS_TO_LOGS,
+ PRINT_STATS_NONE
+} PrintDestination;
+
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsInternal(MemoryContext context, int level,
+ int max_level, int max_children,
+ MemoryContextCounters *totals,
+ PrintDestination print_location,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 83228cfca29..ae17d028ed3 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -232,3 +232,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..d0917b6868e 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e4a9ec65ab4..e456af77206 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1680,6 +1680,9 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-10-07 00:51 torikoshia <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: torikoshia @ 2025-10-07 00:51 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: pgsql-hackers; Daniel Gustafsson <[email protected]>
On 2025-08-20 06:42, Rahila Syed wrote:
> PFA the fix.
Thanks for updating the patch!
Specifying a very small timeout value (such as 0 or 0.0001) and
repeatedly executing the function seems to cause unexpected behavior. In
some cases, it even leads to a crash.
For example:
(session1)=# select pg_backend_pid();
pg_backend_pid
----------------
50917
(session2)=# select pg_get_process_memory_contexts(50917, true,
0.0001);
pg_get_process_memory_contexts
--------------------------------
(0 rows)
(session2)=# \watch 0.01
pg_get_process_memory_contexts
--------------------------------
(,,???,,0,0,0,0,0,0,0)
...
(21 rows)
(session2)=# \watch 0.01
pg_get_process_memory_contexts
--------------------------------
(0 rows)
...
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
This issue occurs on my M1 Mac, but I couldn’t reproduce it on Ubuntu,
so it might be environment-dependent.
Looking at the logs, Assert() is failing:
2025-10-07 08:48:26.766 JST [local] psql [23626] WARNING: 01000:
server process 23646 is processing previous request
2025-10-07 08:48:26.766 JST [local] psql [23626] LOCATION:
pg_get_process_memory_contexts, mcxtfuncs.c:476
TRAP: failed Assert("victim->magic == FREE_PAGE_SPAN_LEADER_MAGIC"),
File: "freepage.c", Line: 1379, PID: 23626
0 postgres 0x000000010357fdf4
ExceptionalCondition + 216
1 postgres 0x00000001035cbe18
FreePageManagerGetInternal + 684
2 postgres 0x00000001035cbb18
FreePageManagerGet + 40
3 postgres 0x00000001035c84cc
dsa_allocate_extended + 788
4 postgres 0x0000000103453af0
pg_get_process_memory_contexts + 992
5 postgres 0x0000000103007e94
ExecMakeFunctionResultSet + 616
6 postgres 0x00000001030506b8
ExecProjectSRF + 304
7 postgres 0x0000000103050434
ExecProjectSet + 268
8 postgres 0x0000000103003270
ExecProcNodeFirst + 92
9 postgres 0x0000000102ffa398
ExecProcNode + 60
10 postgres 0x0000000102ff5050 ExecutePlan
+ 244
11 postgres 0x0000000102ff4ee0
standard_ExecutorRun + 456
12 postgres 0x0000000102ff4d08 ExecutorRun
+ 84
13 postgres 0x0000000103341c84
PortalRunSelect + 296
14 postgres 0x0000000103341694 PortalRun +
656
15 postgres 0x000000010333c4bc
exec_simple_query + 1388
16 postgres 0x000000010333b5d0
PostgresMain + 3252
17 postgres 0x0000000103332750
BackendInitialize + 0
18 postgres 0x0000000103209e48
postmaster_child_launch + 456
19 postgres 0x00000001032118c8
BackendStartup + 304
20 postgres 0x000000010320f72c ServerLoop
+ 372
21 postgres 0x000000010320e1e4
PostmasterMain + 6448
22 postgres 0x0000000103094b0c main + 924
23 dyld 0x0000000199dc2b98 start +
6076
Could you please check if you can reproduce this crash on your
environment?
And a few minor comments on the patch itself:
> + <parameter>stats_timestamp</parameter>
> <type>timestamptz</type> )
As discussed earlier, I believe we decided to remove stats_timestamp,
but it seems it’s still mentioned here.
> + * Update timestamp and signal all the waiting client backends after
> copying
> + * all the statistics.
> + */
> +static void
> +end_memorycontext_reporting(MemoryStatsDSHashEntry *entry,
> MemoryContext oldcontext, HTAB *context_id_lookup)
Should “Update timestamp” in this comment also be removed for
consistency?
The column order differs slightly from pg_backend_memory_contexts.
If there’s no strong reason for the difference, perhaps aligning the
order might improve consistency:
=# select * from pg_get_process_memory_contexts(pg_backend_pid(),
true, 1) ;
name | TopMemoryContext
ident | [NULL]
type | AllocSet
path | {1}
level | 1
total_bytes | 222400
=# select * from pg_backend_memory_contexts;
name | TopMemoryContext
ident | [NULL]
type | AllocSet
level | 1
path | {1}
total_bytes | 99232
...
Regards,
--
Atsushi Torikoshi
Seconded from NTT DATA Japan Corporation to SRA OSS K.K.
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-10-09 08:43 Rahila Syed <[email protected]>
parent: torikoshia <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-10-09 08:43 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi Torikoshia,
Thank you for testing and reviewing the patch.
>
> This issue occurs on my M1 Mac, but I couldn’t reproduce it on Ubuntu,
> so it might be environment-dependent.
> Looking at the logs, Assert() is failing:
>
> 2025-10-07 08:48:26.766 JST [local] psql [23626] WARNING: 01000:
> server process 23646 is processing previous request
> 2025-10-07 08:48:26.766 JST [local] psql [23626] LOCATION:
> pg_get_process_memory_contexts, mcxtfuncs.c:476
> TRAP: failed Assert("victim->magic == FREE_PAGE_SPAN_LEADER_MAGIC"),
> File: "freepage.c", Line: 1379, PID: 23626
> 0 postgres 0x000000010357fdf4
> ExceptionalCondition + 216
> 1 postgres 0x00000001035cbe18
> FreePageManagerGetInternal + 684
> 2 postgres 0x00000001035cbb18
> FreePageManagerGet + 40
> 3 postgres 0x00000001035c84cc
> dsa_allocate_extended + 788
> 4 postgres 0x0000000103453af0
> pg_get_process_memory_contexts + 992
> 5 postgres 0x0000000103007e94
> ExecMakeFunctionResultSet + 616
> 6 postgres 0x00000001030506b8
> ExecProjectSRF + 304
> 7 postgres 0x0000000103050434
> ExecProjectSet + 268
> 8 postgres 0x0000000103003270
> ExecProcNodeFirst + 92
> 9 postgres 0x0000000102ffa398
> ExecProcNode + 60
> 10 postgres 0x0000000102ff5050 ExecutePlan
> + 244
> 11 postgres 0x0000000102ff4ee0
> standard_ExecutorRun + 456
> 12 postgres 0x0000000102ff4d08 ExecutorRun
> + 84
> 13 postgres 0x0000000103341c84
> PortalRunSelect + 296
> 14 postgres 0x0000000103341694 PortalRun +
> 656
> 15 postgres 0x000000010333c4bc
> exec_simple_query + 1388
> 16 postgres 0x000000010333b5d0
> PostgresMain + 3252
> 17 postgres 0x0000000103332750
> BackendInitialize + 0
> 18 postgres 0x0000000103209e48
> postmaster_child_launch + 456
> 19 postgres 0x00000001032118c8
> BackendStartup + 304
> 20 postgres 0x000000010320f72c ServerLoop
> + 372
> 21 postgres 0x000000010320e1e4
> PostmasterMain + 6448
> 22 postgres 0x0000000103094b0c main + 924
> 23 dyld 0x0000000199dc2b98 start +
> 6076
>
>
> Could you please check if you can reproduce this crash on your
> environment?
>
>
I haven't been able to reproduce this issue on Ubuntu. A colleague also
tested it on their Mac
and didn't encounter the problem. I do have a fix in this area that I
believe should address an edge
case where data might be written to freed DSA memory.
Kindly test using the v35 patch and let me know if you still see the issue.
>
> As discussed earlier, I believe we decided to remove stats_timestamp,
> but it seems it’s still mentioned here.
>
>
>
Fixed.
> > + * Update timestamp and signal all the waiting client backends after
> > copying
> > + * all the statistics.
> > + */
> > +static void
> > +end_memorycontext_reporting(MemoryStatsDSHashEntry *entry,
> > MemoryContext oldcontext, HTAB *context_id_lookup)
>
> Should “Update timestamp” in this comment also be removed for
> consistency?
>
>
Fixed.
>
> The column order differs slightly from pg_backend_memory_contexts.
> If there’s no strong reason for the difference, perhaps aligning the
> order might improve consistency:
>
>
Makes sense. I will fix this in the next iteration of the patch.
I am also attaching a test which implements crash testing using injection
points.
I plan to improve the tests further to increase the test coverage of the
feature.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] 0001-v35-0001-Add-pg_get_process_memory_context-function.patch (61.7K, ../../CAH2L28tACFygy9oXbTA8pPQ6KE8uDiO9CgAhku7ERNpdj1YRTw@mail.gmail.com/3-0001-v35-0001-Add-pg_get_process_memory_context-function.patch)
download | inline diff:
From 57a3627b90b4e05d5d15ddef1edad280296120ab Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Tue, 7 Oct 2025 19:38:53 +0530
Subject: [PATCH 1/2] Add pg_get_process_memory_context function
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes the caller specifies
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned.
---
doc/src/sgml/func/func-admin.sgml | 163 ++++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 907 +++++++++++++++++-
src/backend/utils/adt/pg_locale.c | 1 -
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mb/mbutils.c | 1 -
src/backend/utils/mmgr/mcxt.c | 71 +-
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 92 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 3 +
27 files changed, 1296 insertions(+), 41 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 1b465bc8ba7..a9d60202a57 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,136 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type>, <parameter>timeout</parameter> <type>float</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ Busy processes can delay reporting memory context statistics,
+ <parameter>timeout</parameter> specifies the number of seconds
+ to wait for updated statistics. <parameter>timeout</parameter> can be
+ specified in fractions of a second.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +432,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false, 0.5) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+path | {1}
+level | 1
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 884b6a23817..41d80661320 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -682,6 +682,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean, float) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index fb5d3b27224..16092f619b2 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -790,6 +790,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e84e8663e96..5b3e08805bf 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index ba63b84dfc5..29454b8bf8a 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..ac97a39447c 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -867,6 +867,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index e1f142f20c7..c711c887ef6 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 2fa045e6b0f..fe3d32e40b0 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -51,6 +51,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -150,6 +151,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -343,6 +345,7 @@ CreateOrAttachShmemStructs(void)
WaitEventCustomShmemInit();
InjectionPointShmemInit();
AioShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 96f29aafc39..550a3a77bb8 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -50,6 +50,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index d356830f756..0fdd2202ddd 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3538,6 +3538,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 7553f6eacef..cdbc7309206 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -160,6 +160,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -401,6 +402,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..17b50e853d5 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,13 +15,38 @@
#include "postgres.h"
+#include "access/twophase.h"
+#include "catalog/pg_authid_d.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+#define CLIENT_KEY_SIZE 64
+
+static LWLock *client_keys_lock = NULL;
+static int *client_keys = NULL;
+static dshash_table *MemoryStatsDsHash = NULL;
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts, int max_levels);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
@@ -89,7 +114,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +168,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +183,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +229,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +256,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +264,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +345,824 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to timeout value specified by the user, give up and
+ * return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ double timeout = PG_GETARG_FLOAT8(2);
+ PGPROC *proc;
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+
+ if (timeout < 0)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ (errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),
+ errmsg("\"timeout\" must not be negative")));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is just a warning so a loop-through-resultset will not abort
+ * if one backend terminated on its own during the run.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit(1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ /*
+ * XXX. If the process exits without cleaning up its slot, i.e in case of an
+ * abrupt crash the client_keys slot won't be reset thus resulting in false
+ * negative and WARNING would be thrown in case another process with same
+ * slot index is queried for statistics.
+ */
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request", pid));
+ LWLockRelease(client_keys_lock);
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from.
+ * If this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since the
+ * previous server was notified, it might attempt to read the same client entry
+ * and respond incorrectly with its statistics. By storing the server ID in the
+ * client entry, we prevent any previously signalled server process from writing
+ * its statistics in the space meant for the newly requested process.
+ */
+ entry->server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ PG_TRY();
+ {
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m", pid));
+ memstats_dsa_cleanup(key);
+ PG_RETURN_NULL();
+ }
+
+ while (1)
+ {
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ */
+ if (entry->stats_written)
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Recheck the state of the backend before sleeping on the
+ * condition variable to ensure the process is still alive. Only
+ * check the relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The process ending during memory context processing is not an
+ * error.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ memstats_dsa_cleanup(key);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+
+
+ /*
+ * Wait for the timeout as defined by the user. If no statistics
+ * are available within the allowed time then return NULL. The
+ * timer is defined in milliseconds since that's what the
+ * condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (timeout * 1000), WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ if (memcxt_info[i].name[0] != '\0')
+ {
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+ }
+ else
+ nulls[0] = true;
+
+ if (memcxt_info[i].ident[0] != '\0')
+ {
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ }
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum, path_length, INT4OID);
+ values[3] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[3] = true;
+
+ values[4] = Int32GetDatum(memcxt_info[i].levels);
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ }
+ PG_CATCH();
+ {
+ memstats_dsa_cleanup(key);
+ ConditionVariableCancelSleep();
+ }
+ PG_END_TRY();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in
+ * per-process DSA pointers. These pointers are stored in a dshash table with
+ * key as requesting clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ bool summary = false;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ /* Retreive the client key for publishing statistics */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ LWLockRelease(client_keys_lock);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Entry has been deleted due to client process exit
+ * Make sure that the client always deletes the entry
+ * after taking required lock or this function may end up writing
+ * to unallocated memory.
+ */
+ if (!found)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /*
+ * The client has timed out waiting for us to write statistics and is
+ * requesting statistics from some other process
+ */
+ if (entry->server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ summary = entry->summary;
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1, 100);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &c,
+ HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsInternal(c, 1, 100, 100, &grand_totals,
+ PRINT_STATS_NONE, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts, 100);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1, 100);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name, "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id -
+ num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+}
+
+/*
+ * Clean up before exit from ProcessGetMemoryContextInterrupt
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Empty this processes slot, so other clients can request memory
+ * statistics
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts, int max_levels)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), max_levels);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 97c2ac1faf9..ab768a7a91f 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -45,7 +45,6 @@
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
-#include "utils/relcache.h"
#include "utils/syscache.h"
#ifdef WIN32
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..fb3f2d21fa0 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -662,6 +662,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 886ecbad871..308016d7763 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -39,7 +39,6 @@
#include "mb/pg_wchar.h"
#include "utils/fmgrprotos.h"
#include "utils/memutils.h"
-#include "utils/relcache.h"
#include "varatt.h"
/*
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..3a5422c7273 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -40,6 +40,7 @@
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+#include "utils/hsearch.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/memutils_internal.h"
@@ -176,10 +177,6 @@ MemoryContext PortalContext = NULL;
static void MemoryContextDeleteOnly(MemoryContext context);
static void MemoryContextCallResetCallbacks(MemoryContext context);
-static void MemoryContextStatsInternal(MemoryContext context, int level,
- int max_level, int max_children,
- MemoryContextCounters *totals,
- bool print_to_stderr);
static void MemoryContextStatsPrint(MemoryContext context, void *passthru,
const char *stats_string,
bool print_to_stderr);
@@ -877,11 +874,19 @@ MemoryContextStatsDetail(MemoryContext context,
bool print_to_stderr)
{
MemoryContextCounters grand_totals;
+ int num_contexts;
+ PrintDestination print_location;
memset(&grand_totals, 0, sizeof(grand_totals));
+ if (print_to_stderr)
+ print_location = PRINT_STATS_TO_STDERR;
+ else
+ print_location = PRINT_STATS_TO_LOGS;
+
+ /* num_contexts report number of contexts aggregated in the output */
MemoryContextStatsInternal(context, 1, max_level, max_children,
- &grand_totals, print_to_stderr);
+ &grand_totals, print_location, &num_contexts);
if (print_to_stderr)
fprintf(stderr,
@@ -916,13 +921,14 @@ MemoryContextStatsDetail(MemoryContext context,
* One recursion level for MemoryContextStats
*
* Print stats for this context if possible, but in any case accumulate counts
- * into *totals (if not NULL).
+ * into *totals (if not NULL). The callers should make sure that print_location
+ * is set to PRINT_STATS_TO_STDERR or PRINT_STATS_TO_LOGS or PRINT_STATS_NONE.
*/
-static void
+void
MemoryContextStatsInternal(MemoryContext context, int level,
int max_level, int max_children,
MemoryContextCounters *totals,
- bool print_to_stderr)
+ PrintDestination print_location, int *num_contexts)
{
MemoryContext child;
int ichild;
@@ -930,10 +936,39 @@ MemoryContextStatsInternal(MemoryContext context, int level,
Assert(MemoryContextIsValid(context));
/* Examine the context itself */
- context->methods->stats(context,
- MemoryContextStatsPrint,
- &level,
- totals, print_to_stderr);
+ switch (print_location)
+ {
+ case PRINT_STATS_TO_STDERR:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, true);
+ break;
+
+ case PRINT_STATS_TO_LOGS:
+ context->methods->stats(context,
+ MemoryContextStatsPrint,
+ &level,
+ totals, false);
+ break;
+
+ case PRINT_STATS_NONE:
+
+ /*
+ * Do not print the statistics if print_location is
+ * PRINT_STATS_NONE, only compute totals. This is used in
+ * reporting of memory context statistics via a sql function. Last
+ * parameter is not relevant.
+ */
+ context->methods->stats(context,
+ NULL,
+ NULL,
+ totals, false);
+ break;
+ }
+
+ /* Increment the context count for each of the recursive call */
+ *num_contexts = *num_contexts + 1;
/*
* Examine children.
@@ -953,7 +988,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
MemoryContextStatsInternal(child, level + 1,
max_level, max_children,
totals,
- print_to_stderr);
+ print_location, num_contexts);
}
}
@@ -972,7 +1007,13 @@ MemoryContextStatsInternal(MemoryContext context, int level,
child = MemoryContextTraverseNext(child, context);
}
- if (print_to_stderr)
+ /*
+ * Add the count of children contexts which are traversed in the
+ * non-recursive manner.
+ */
+ *num_contexts = *num_contexts + ichild;
+
+ if (print_location == PRINT_STATS_TO_STDERR)
{
for (int i = 0; i < level; i++)
fprintf(stderr, " ");
@@ -985,7 +1026,7 @@ MemoryContextStatsInternal(MemoryContext context, int level,
local_totals.freechunks,
local_totals.totalspace - local_totals.freespace);
}
- else
+ else if (print_location == PRINT_STATS_TO_LOGS)
ereport(LOG_SERVER_ONLY,
(errhidestmt(true),
errhidecontext(true),
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 7c20180637f..8859ed449d0 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8613,6 +8613,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool float8',
+ proallargtypes => '{int4,bool,float8,text,text,text,_int4,int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, timeout, name, ident, type, path, level, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..1e59a7f910f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 06a1ffd4b08..4c71d756a2d 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -135,3 +135,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7bbe5a36959..38451035fd4 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,10 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "storage/condition_variable.h"
+#include "storage/lmgr.h"
+#include "utils/dsa.h"
+#include "lib/dshash.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,6 +51,23 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 64
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum size per context. Actual size may be lower as this assumes the worst
+ * case of deepest path and longest identifiers (name and ident, thus the
+ * multiplication by 2). The path depth is limited to 100 like for memory
+ * context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
/*
* Standard top-level memory contexts.
@@ -319,4 +339,74 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+/* Dynamic shared memory state for statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * Used for storage of transient identifiers for pg_get_backend_memory_contexts
+ */
+typedef struct MemoryStatsContextId
+{
+ MemoryContext context;
+ int context_id;
+} MemoryStatsContextId;
+
+/*
+ * This is passed to MemoryContextStatsInternal to determine whether
+ * to print context statistics or not and where to print them logs or
+ * stderr.
+ */
+typedef enum PrintDestination
+{
+ PRINT_STATS_TO_STDERR = 0,
+ PRINT_STATS_TO_LOGS,
+ PRINT_STATS_NONE
+} PrintDestination;
+
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsInternal(MemoryContext context, int level,
+ int max_level, int max_children,
+ MemoryContextCounters *totals,
+ PrintDestination print_location,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 83228cfca29..ae17d028ed3 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -232,3 +232,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..d0917b6868e 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false, 20)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 37f26f6c6b7..63fd382d4a0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1680,6 +1680,9 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] 0002-Test-fixes.patch (8.0K, ../../CAH2L28tACFygy9oXbTA8pPQ6KE8uDiO9CgAhku7ERNpdj1YRTw@mail.gmail.com/4-0002-Test-fixes.patch)
download | inline diff:
From b537ccf2b5f56eedcb22f066e00a6d61aececa9e Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 9 Oct 2025 12:07:20 +0530
Subject: [PATCH] Test fixes
---
src/backend/utils/adt/mcxtfuncs.c | 2 +
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 32 +++++++++
.../t/001_memcontext_inj.pl | 45 ++++++++++++
.../test_memcontext_reporting--1.0.sql | 7 ++
.../test_memcontext_reporting.c | 68 +++++++++++++++++++
.../test_memcontext_reporting.control | 4 ++
7 files changed, 159 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index aa4e0a2e670..8c3d3f321aa 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -27,6 +27,7 @@
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
#include "utils/memutils.h"
#include "utils/wait_event_types.h"
@@ -541,6 +542,7 @@ pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_NULL();
}
+ INJECTION_POINT("memcontext-client-crash", NULL);
while (1)
{
entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 902a7954101..a31a2578c18 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -31,6 +31,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..01a7baa0263
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,32 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+EXTENSION = test_memcontext_reporting
+DATA = test_memcontext_reporting--1.0.sql
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..842f32376fd
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,45 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing enabling data checksums in an online cluster,
+# comprising of a primary and a replicated standby, with concurrent activity
+# via pgbench runs
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init(allows_streaming => 1, no_data_checksums => 1);
+# max_connections need to be bumped in order to accommodate for pgbench clients
+# and log_statement is dialled down since it otherwise will generate enormous
+# amounts of logging. Page verification failures are still logged.
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION test_memcontext_reporting;');
+# Attaching to an injection point that crashes memory context client
+$node->safe_psql('postgres', 'SELECT memcontext_crash_client();');
+
+my $pid = $node->safe_psql('postgres', "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+#Client should have crashed
+$node->safe_psql('postgres', "select pg_get_process_memory_contexts($pid, true, 5);");
+print "PID";
+print $pid;
+#Query the same process for memory context using some other client and it should succeed.
+my $topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true, 5) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
new file mode 100644
index 00000000000..7f628bf24e2
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
@@ -0,0 +1,7 @@
+CREATE FUNCTION memcontext_crash_server()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION memcontext_crash_client()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..f77875b437f
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,68 @@
+/*
+ * -------------------------------------------------------------------------
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include "utils/injection_point.h"
+#include "funcapi.h"
+#include "utils/injection_point.h"
+
+PG_MODULE_MAGIC;
+
+void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
+
+/*
+ * memcontext_crash_client
+ *
+ * Ensure that the client process aborts in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_client);
+Datum
+memcontext_crash_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-client-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_crash_server
+ *
+ * Ensure that the server process crashes in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_server);
+Datum
+memcontext_crash_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-server-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-10-14 09:00 torikoshia <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: torikoshia @ 2025-10-14 09:00 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: pgsql-hackers; Daniel Gustafsson <[email protected]>
On 2025-10-09 17:43, Rahila Syed wrote:
> Hi Torikoshia,
>
> Thank you for testing and reviewing the patch.
>
>> This issue occurs on my M1 Mac, but I couldn’t reproduce it on
>> Ubuntu,
>> so it might be environment-dependent.
>
>> Looking at the logs, Assert() is failing:
>>
>> 2025-10-07 08:48:26.766 JST [local] psql [23626] WARNING: 01000:
>>
>> server process 23646 is processing previous request
>> 2025-10-07 08:48:26.766 JST [local] psql [23626] LOCATION:
>> pg_get_process_memory_contexts, mcxtfuncs.c:476
>> TRAP: failed Assert("victim->magic ==
>> FREE_PAGE_SPAN_LEADER_MAGIC"),
>> File: "freepage.c", Line: 1379, PID: 23626
>> 0 postgres 0x000000010357fdf4
>> ExceptionalCondition + 216
>> 1 postgres 0x00000001035cbe18
>> FreePageManagerGetInternal + 684
>> 2 postgres 0x00000001035cbb18
>> FreePageManagerGet + 40
>> 3 postgres 0x00000001035c84cc
>> dsa_allocate_extended + 788
>> 4 postgres 0x0000000103453af0
>> pg_get_process_memory_contexts + 992
>> 5 postgres 0x0000000103007e94
>> ExecMakeFunctionResultSet + 616
>> 6 postgres 0x00000001030506b8
>> ExecProjectSRF + 304
>> 7 postgres 0x0000000103050434
>> ExecProjectSet + 268
>> 8 postgres 0x0000000103003270
>> ExecProcNodeFirst + 92
>> 9 postgres 0x0000000102ffa398
>> ExecProcNode + 60
>> 10 postgres 0x0000000102ff5050
>> ExecutePlan
>> + 244
>> 11 postgres 0x0000000102ff4ee0
>> standard_ExecutorRun + 456
>> 12 postgres 0x0000000102ff4d08
>> ExecutorRun
>> + 84
>> 13 postgres 0x0000000103341c84
>> PortalRunSelect + 296
>> 14 postgres 0x0000000103341694
>> PortalRun +
>> 656
>> 15 postgres 0x000000010333c4bc
>> exec_simple_query + 1388
>> 16 postgres 0x000000010333b5d0
>> PostgresMain + 3252
>> 17 postgres 0x0000000103332750
>> BackendInitialize + 0
>> 18 postgres 0x0000000103209e48
>> postmaster_child_launch + 456
>> 19 postgres 0x00000001032118c8
>> BackendStartup + 304
>> 20 postgres 0x000000010320f72c
>> ServerLoop
>> + 372
>> 21 postgres 0x000000010320e1e4
>> PostmasterMain + 6448
>> 22 postgres 0x0000000103094b0c main +
>> 924
>> 23 dyld 0x0000000199dc2b98 start
>> +
>> 6076
>>
>> Could you please check if you can reproduce this crash on your
>> environment?
>
> I haven't been able to reproduce this issue on Ubuntu. A colleague
> also tested it on their Mac
> and didn't encounter the problem. I do have a fix in this area that I
> believe should address an edge
> case where data might be written to freed DSA memory.
>
> Kindly test using the v35 patch and let me know if you still see the
> issue.
Thanks for the update.
v35 works fine on my environment.
I ran the same test and haven’t encountered the crash anymore.
The addition of the following code appears to have resolved the issue:
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
Since you seem to make a next version patch, I understand v35 is an
interim patch,
so this isn’t a major concern, but I encountered trailing whitespace
warnings when applying the patches:
$ git apply
0001-v35-0001-Add-pg_get_process_memory_context-function.patch
0001-v35-0001-Add-pg_get_process_memory_context-function.patch:705:
trailing whitespace.
0001-v35-0001-Add-pg_get_process_memory_context-function.patch:1066:
trailing whitespace.
--
Regards,
--
Atsushi Torikoshi
Seconded from NTT DATA Japan Corporation to SRA OSS K.K.
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-10-28 05:36 Rahila Syed <[email protected]>
parent: torikoshia <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-10-28 05:36 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: pgsql-hackers; Daniel Gustafsson <[email protected]>
Hi,
PFA an updated v39 patch which is ready for review in the upcoming
commitfest.
> v35 works fine on my environment.
> I ran the same test and haven’t encountered the crash anymore.
>
Thank you for testing and confirming the fix.
> The addition of the following code appears to have resolved the issue:
>
> +memstats_dsa_cleanup(char *key)
> +{
> + MemoryStatsDSHashEntry *entry;
> +
> + entry = dshash_find(MemoryStatsDsHash, key, true);
>
Yes, without this code, the dsa memory was being freed in the timeout path
without acquiring a lock.
Since you seem to make a next version patch, I understand v35 is an
> interim patch,
> so this isn’t a major concern, but I encountered trailing whitespace
> warnings when applying the patches.
>
$ git apply
> 0001-v35-0001-Add-pg_get_process_memory_context-function.patch
> 0001-v35-0001-Add-pg_get_process_memory_context-function.patch:705:
> trailing whitespace.
> 0001-v35-0001-Add-pg_get_process_memory_context-function.patch:1066:
> trailing whitespace.
>
>
Thanks, should be fixed now.
The updated patch contains the following changes. These changes are
addressing some review comments
discussed off list and a couple of bugs found while doing injection points
tests.
1.
All the changes made to MemoryContextStatsInternal and
MemoryContextStatsDetail are removed.
Instead of modifying these functions, I have written a separate function
MemoryContextStatsCounter
that takes care of counting statistics. This approach ensures that the
existing functions remain unchanged.
2. Changes to ensure that the wait loop does not exceed the prescribed wait
time.
Additional exit condition has been added to the infinite loop that waits
for request completion.
This allows the pg_get_memoy_context_statistics function to return if the
elapsed time goes beyond
a set limit i.e the following timeout.
3. The user facing timeout is removed as that would complicate the user
interface. CFIs
are called frequently and the requests are likely to be addressed promptly.
A predefined macro MEMORY_CONTEXT_STATS_TIMEOUT 5 (secs) is used for
timeout
instead. This would also remove the possibility of a user setting very low
timeouts, which
could cause requests to be incomplete and result in NULL outputs.
4. Miscellaneous cleanups to improve comments and remove left over comments
from older
versions. Also, removed an unnecessary argument from the
PublishMemoryContext function.
5. Addressed Torikoshias suggestion to change the order of columns to match
pg_backend_memory_contexts.
6. Attached is a test module that tests error handling by introducing
errors using
injection points. I have resolved a few bugs, so the memory monitoring
function
now runs correctly after the previous request ended with an error.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v39-0001-Add-function-to-report-memory-context-statistics.patch (57.7K, ../../CAH2L28t=c_iBj0+LoKX0TSTrNJB2nTgH6MPD7QgRb8kSBZ7HVg@mail.gmail.com/3-v39-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From 2bd097622985e2c800ee0ca00c9bb08887ea66f7 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 23 Oct 2025 17:31:52 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 157 +++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 910 +++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 29 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 81 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 3 +
25 files changed, 1255 insertions(+), 24 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 1b465bc8ba7..a5c66837241 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,130 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +426,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 823776c1498..3bb2a9d3c9b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -692,6 +692,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 5084af7dfb6..8bf6f6eb743 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e84e8663e96..5b3e08805bf 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index ba63b84dfc5..29454b8bf8a 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 78e39e5f866..ac97a39447c 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -867,6 +867,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index e1f142f20c7..c711c887ef6 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 2fa045e6b0f..fe3d32e40b0 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -51,6 +51,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -150,6 +151,7 @@ CalculateShmemSize(int *num_semaphores)
size = add_size(size, InjectionPointShmemSize());
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -343,6 +345,7 @@ CreateOrAttachShmemStructs(void)
WaitEventCustomShmemInit();
InjectionPointShmemInit();
AioShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 96f29aafc39..550a3a77bb8 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -50,6 +50,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 7dd75a490aa..e726f40dfbb 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3539,6 +3539,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 7553f6eacef..cdbc7309206 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -160,6 +160,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -401,6 +402,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..a62f3d6dc93 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,20 +15,51 @@
#include "postgres.h"
+#include "access/twophase.h"
+#include "catalog/pg_authid_d.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+#define CLIENT_KEY_SIZE 64
+
+static LWLock *client_keys_lock = NULL;
+static int *client_keys = NULL;
+static dshash_table *MemoryStatsDsHash = NULL;
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
* Used for storage of transient identifiers for
@@ -89,7 +120,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +174,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +189,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +235,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +262,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +270,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +351,821 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give up
+ * and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is a warning because we don't want to break loops.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ /*
+ * XXX. If the process exits without cleaning up its slot, i.e in case of
+ * an abrupt crash the client_keys slot won't be reset thus resulting in
+ * false negative and WARNING would be thrown in case another process with
+ * same slot index is queried for statistics.
+ */
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request", pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m", pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ INJECTION_POINT("memcontext-client-crash", NULL);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ INJECTION_POINT("memcontext-client-crash", NULL);
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000), WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum, path_length, INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in
+ * per-process DSA pointers. These pointers are stored in a dshash table with
+ * key as requesting clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ bool summary = false;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Retreive the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process
+ */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ INJECTION_POINT("memcontext-server-crash", NULL);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Entry has been deleted due to client process exit. Make sure that the
+ * client always deletes the entry after taking required lock or this
+ * function may end up writing to unallocated memory.
+ */
+ if (!found)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /*
+ * The client has timed out waiting for us to write statistics and is
+ * requesting statistics from some other process
+ */
+ if (entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+ summary = entry->summary;
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &c,
+ HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name, "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id -
+ num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+}
+
+/*
+ * Clean up before exit from ProcessGetMemoryContextInterrupt
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+/* Used for testing purposes */
+dsa_area *
+pg_get_memstats_dsa_area(void)
+{
+ if (MemoryStatsDsaArea != NULL)
+ return MemoryStatsDsaArea;
+ else
+ return NULL;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 641e535a73c..fb3f2d21fa0 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -662,6 +662,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..56c2048c67a 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1008,6 +1008,35 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of children contexts which are traversed
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index b51d2b17379..52e8e525d86 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8617,6 +8617,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 1bef98471c3..1e59a7f910f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 06a1ffd4b08..4c71d756a2d 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -135,3 +135,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7bbe5a36959..2d7220cde45 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,10 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "storage/condition_variable.h"
+#include "storage/lmgr.h"
+#include "utils/dsa.h"
+#include "lib/dshash.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,6 +51,26 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum size per context statistics. The identifier and name are statically
+ * allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
/*
* Standard top-level memory contexts.
@@ -149,6 +172,7 @@ extern MemoryContext BumpContextCreate(MemoryContext parent,
Size minContextSize,
Size initBlockSize,
Size maxBlockSize);
+extern dsa_area *pg_get_memstats_dsa_area(void);
/*
* Recommended default alloc parameters, suitable for "ordinary" contexts
@@ -319,4 +343,59 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+/* Dynamic shared memory state for statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * Used for storage of transient identifiers for pg_get_backend_memory_contexts
+ */
+typedef struct MemoryStatsContextId
+{
+ MemoryContext context;
+ int context_id;
+} MemoryStatsContextId;
+
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3b37fafa65b..21c65ad2d10 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..c9da4fc8c90 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 377a7946585..e9a7adfd380 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1682,6 +1682,9 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] v2-0002-Test-module-to-test-memory-context-reporting-with-in.patch (9.5K, ../../CAH2L28t=c_iBj0+LoKX0TSTrNJB2nTgH6MPD7QgRb8kSBZ7HVg@mail.gmail.com/4-v2-0002-Test-module-to-test-memory-context-reporting-with-in.patch)
download | inline diff:
From e21832c610574f48496c961a7318070b720568de Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 23 Oct 2025 18:01:36 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 32 +++++
.../t/001_memcontext_inj.pl | 58 +++++++++
.../test_memcontext_reporting--1.0.sql | 11 ++
.../test_memcontext_reporting.c | 123 ++++++++++++++++++
.../test_memcontext_reporting.control | 4 +
6 files changed, 229 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 902a7954101..a31a2578c18 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -31,6 +31,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..01a7baa0263
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,32 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+EXTENSION = test_memcontext_reporting
+DATA = test_memcontext_reporting--1.0.sql
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..69d8489eb37
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,58 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init(allows_streaming => 1);
+# max_connections need to be bumped in order to accommodate for pgbench clients
+# and log_statement is dialled down since it otherwise will generate enormous
+# amounts of logging. Page verification failures are still logged.
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION test_memcontext_reporting;');
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+# Attaching to a client process injection point that throws an error
+$node->safe_psql('postgres', "select injection_points_attach('memcontext-client-crash', 'error');");
+
+my $pid = $node->safe_psql('postgres', "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+print "PID";
+print $pid;
+
+#Client should have thrown error
+$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+like ( $psql_err, qr/error triggered for injection point memcontext-client-crash/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres', "select injection_points_detach('memcontext-client-crash');");
+my $topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres', "select injection_points_attach('memcontext-server-crash', 'error');");
+
+#Server should have thrown error
+$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres', "select injection_points_detach('memcontext-server-crash');");
+$topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
new file mode 100644
index 00000000000..181daf429d0
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
@@ -0,0 +1,11 @@
+CREATE FUNCTION memcontext_crash_server()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION memcontext_crash_client()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION dsa_dump_sql()
+RETURNS bigint
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..955155524c2
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,123 @@
+/*
+ * -------------------------------------------------------------------------
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include "utils/injection_point.h"
+#include "funcapi.h"
+#include "utils/injection_point.h"
+#include "storage/dsm_registry.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
+
+/*
+ * memcontext_crash_client
+ *
+ * Ensure that the client process aborts in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_client);
+Datum
+memcontext_crash_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-client-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(memcontext_detach_client);
+Datum
+memcontext_detach_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointDetach("memcontext-client-crash");
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_crash_server
+ *
+ * Ensure that the server process crashes in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_server);
+Datum
+memcontext_crash_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-server-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_detach_server
+ *
+ * Detach the injection point which crashes the server
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_detach_server);
+Datum
+memcontext_detach_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointDetach("memcontext-server-crash");
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * dsa_dump_sql
+ */
+PG_FUNCTION_INFO_V1(dsa_dump_sql);
+Datum
+dsa_dump_sql(PG_FUNCTION_ARGS)
+{
+ bool found;
+ size_t tot_size;
+ dsa_area *memstats_dsa_area;
+
+ memstats_dsa_area = pg_get_memstats_dsa_area();
+
+ if (memstats_dsa_area == NULL)
+ memstats_dsa_area = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ tot_size = dsa_get_total_size(memstats_dsa_area);
+ dsa_detach(memstats_dsa_area);
+ PG_RETURN_INT64(tot_size);
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-11-07 22:55 Rahila Syed <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-11-07 22:55 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>
Hi,
I have attached a version 40 patch that has been rebased onto the
latest master branch, as CFbot indicated a rebase was needed.
The test module patch is unchanged.
Thank you,
Rahila Syed
On Tue, Oct 28, 2025 at 11:06 AM Rahila Syed <[email protected]> wrote:
> Hi,
>
> PFA an updated v39 patch which is ready for review in the upcoming
> commitfest.
>
>
>> v35 works fine on my environment.
>> I ran the same test and haven’t encountered the crash anymore.
>>
>
> Thank you for testing and confirming the fix.
>
>
>> The addition of the following code appears to have resolved the issue:
>>
>> +memstats_dsa_cleanup(char *key)
>> +{
>> + MemoryStatsDSHashEntry *entry;
>> +
>> + entry = dshash_find(MemoryStatsDsHash, key, true);
>>
>
> Yes, without this code, the dsa memory was being freed in the timeout path
> without acquiring a lock.
>
> Since you seem to make a next version patch, I understand v35 is an
>> interim patch,
>> so this isn’t a major concern, but I encountered trailing whitespace
>> warnings when applying the patches.
>>
> $ git apply
>> 0001-v35-0001-Add-pg_get_process_memory_context-function.patch
>> 0001-v35-0001-Add-pg_get_process_memory_context-function.patch:705:
>> trailing whitespace.
>> 0001-v35-0001-Add-pg_get_process_memory_context-function.patch:1066:
>> trailing whitespace.
>>
>>
> Thanks, should be fixed now.
>
> The updated patch contains the following changes. These changes are
> addressing some review comments
> discussed off list and a couple of bugs found while doing injection points
> tests.
>
> 1.
> All the changes made to MemoryContextStatsInternal and
> MemoryContextStatsDetail are removed.
> Instead of modifying these functions, I have written a separate function
> MemoryContextStatsCounter
> that takes care of counting statistics. This approach ensures that the
> existing functions remain unchanged.
>
> 2. Changes to ensure that the wait loop does not exceed the prescribed
> wait time.
> Additional exit condition has been added to the infinite loop that waits
> for request completion.
> This allows the pg_get_memoy_context_statistics function to return if the
> elapsed time goes beyond
> a set limit i.e the following timeout.
>
> 3. The user facing timeout is removed as that would complicate the user
> interface. CFIs
> are called frequently and the requests are likely to be addressed promptly.
> A predefined macro MEMORY_CONTEXT_STATS_TIMEOUT 5 (secs) is used for
> timeout
> instead. This would also remove the possibility of a user setting very
> low timeouts, which
> could cause requests to be incomplete and result in NULL outputs.
>
> 4. Miscellaneous cleanups to improve comments and remove left over
> comments from older
> versions. Also, removed an unnecessary argument from the
> PublishMemoryContext function.
>
> 5. Addressed Torikoshias suggestion to change the order of columns to match
> pg_backend_memory_contexts.
>
> 6. Attached is a test module that tests error handling by introducing
> errors using
> injection points. I have resolved a few bugs, so the memory monitoring
> function
> now runs correctly after the previous request ended with an error.
>
> Thank you,
> Rahila Syed
>
Attachments:
[application/octet-stream] v40-0001-Add-function-to-report-memory-context-statistics.patch (57.7K, ../../CAH2L28v5qXPpfmVLFDcMFL4MMqsKP92FXhW2eosHA5LbatKnrw@mail.gmail.com/3-v40-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From c8cddf59c3d9b1f6967acd86ec79945bac06c2f0 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Sat, 8 Nov 2025 04:06:21 +0530
Subject: [PATCH] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 157 +++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 910 +++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 29 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 81 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 3 +
25 files changed, 1255 insertions(+), 24 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 1b465bc8ba7..a5c66837241 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,130 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +426,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 059e8778ca7..c63fd6783bd 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -692,6 +692,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index ed19c74bb19..34bdb88fa7f 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e84e8663e96..5b3e08805bf 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index ba63b84dfc5..29454b8bf8a 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ce6b5299324..fdd385e492d 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -871,6 +871,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c4a888a081c..00f03b36ed8 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index b23d0c19360..a5ed58a18c5 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -140,6 +141,7 @@ CalculateShmemSize(void)
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
InjectionPointShmemInit();
AioShmemInit();
WaitLSNShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 1504fafe6d8..c5e69151756 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 2bd89102686..da8f2b97986 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3549,6 +3549,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index c1ac71ff7f2..644d8d988e1 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -162,6 +162,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -404,6 +405,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..a62f3d6dc93 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,20 +15,51 @@
#include "postgres.h"
+#include "access/twophase.h"
+#include "catalog/pg_authid_d.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
+#include "utils/acl.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+#define CLIENT_KEY_SIZE 64
+
+static LWLock *client_keys_lock = NULL;
+static int *client_keys = NULL;
+static dshash_table *MemoryStatsDsHash = NULL;
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
* Used for storage of transient identifiers for
@@ -89,7 +120,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +174,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +189,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +235,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +262,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +270,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +351,821 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give up
+ * and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is a warning because we don't want to break loops.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ /*
+ * XXX. If the process exits without cleaning up its slot, i.e in case of
+ * an abrupt crash the client_keys slot won't be reset thus resulting in
+ * false negative and WARNING would be thrown in case another process with
+ * same slot index is queried for statistics.
+ */
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request", pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m", pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ INJECTION_POINT("memcontext-client-crash", NULL);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ INJECTION_POINT("memcontext-client-crash", NULL);
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000), WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ }
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum, path_length, INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in
+ * per-process DSA pointers. These pointers are stored in a dshash table with
+ * key as requesting clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ bool summary = false;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Retreive the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process
+ */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
+
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ INJECTION_POINT("memcontext-server-crash", NULL);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Entry has been deleted due to client process exit. Make sure that the
+ * client always deletes the entry after taking required lock or this
+ * function may end up writing to unallocated memory.
+ */
+ if (!found)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /*
+ * The client has timed out waiting for us to write statistics and is
+ * requesting statistics from some other process
+ */
+ if (entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+ summary = entry->summary;
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &c,
+ HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name, "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id -
+ num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+}
+
+/*
+ * Clean up before exit from ProcessGetMemoryContextInterrupt
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+/* Used for testing purposes */
+dsa_area *
+pg_get_memstats_dsa_area(void)
+{
+ if (MemoryStatsDsaArea != NULL)
+ return MemoryStatsDsaArea;
+ else
+ return NULL;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 98f9598cd78..202403ebc63 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -658,6 +658,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..56c2048c67a 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1008,6 +1008,35 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of children contexts which are traversed
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 5cf9e12fcb9..bb72b85457d 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8617,6 +8617,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 9a7d733ddef..b76f24baed6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 5b0ce383408..613e769c84e 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -136,3 +136,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7bbe5a36959..2d7220cde45 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,10 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "storage/condition_variable.h"
+#include "storage/lmgr.h"
+#include "utils/dsa.h"
+#include "lib/dshash.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,6 +51,26 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum size per context statistics. The identifier and name are statically
+ * allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
/*
* Standard top-level memory contexts.
@@ -149,6 +172,7 @@ extern MemoryContext BumpContextCreate(MemoryContext parent,
Size minContextSize,
Size initBlockSize,
Size maxBlockSize);
+extern dsa_area *pg_get_memstats_dsa_area(void);
/*
* Recommended default alloc parameters, suitable for "ordinary" contexts
@@ -319,4 +343,59 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+/* Dynamic shared memory state for statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * Used for storage of transient identifiers for pg_get_backend_memory_contexts
+ */
+typedef struct MemoryStatsContextId
+{
+ MemoryContext context;
+ int context_id;
+} MemoryStatsContextId;
+
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3b37fafa65b..21c65ad2d10 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..c9da4fc8c90 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 432509277c9..2990c807f45 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1684,6 +1684,9 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] v2-0002-Test-module-to-test-memory-context-reporting-with-in.patch (9.5K, ../../CAH2L28v5qXPpfmVLFDcMFL4MMqsKP92FXhW2eosHA5LbatKnrw@mail.gmail.com/4-v2-0002-Test-module-to-test-memory-context-reporting-with-in.patch)
download | inline diff:
From e21832c610574f48496c961a7318070b720568de Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 23 Oct 2025 18:01:36 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 32 +++++
.../t/001_memcontext_inj.pl | 58 +++++++++
.../test_memcontext_reporting--1.0.sql | 11 ++
.../test_memcontext_reporting.c | 123 ++++++++++++++++++
.../test_memcontext_reporting.control | 4 +
6 files changed, 229 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 902a7954101..a31a2578c18 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -31,6 +31,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..01a7baa0263
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,32 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+EXTENSION = test_memcontext_reporting
+DATA = test_memcontext_reporting--1.0.sql
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..69d8489eb37
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,58 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init(allows_streaming => 1);
+# max_connections need to be bumped in order to accommodate for pgbench clients
+# and log_statement is dialled down since it otherwise will generate enormous
+# amounts of logging. Page verification failures are still logged.
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION test_memcontext_reporting;');
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+# Attaching to a client process injection point that throws an error
+$node->safe_psql('postgres', "select injection_points_attach('memcontext-client-crash', 'error');");
+
+my $pid = $node->safe_psql('postgres', "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+print "PID";
+print $pid;
+
+#Client should have thrown error
+$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+like ( $psql_err, qr/error triggered for injection point memcontext-client-crash/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres', "select injection_points_detach('memcontext-client-crash');");
+my $topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres', "select injection_points_attach('memcontext-server-crash', 'error');");
+
+#Server should have thrown error
+$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres', "select injection_points_detach('memcontext-server-crash');");
+$topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
new file mode 100644
index 00000000000..181daf429d0
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
@@ -0,0 +1,11 @@
+CREATE FUNCTION memcontext_crash_server()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION memcontext_crash_client()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION dsa_dump_sql()
+RETURNS bigint
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..955155524c2
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,123 @@
+/*
+ * -------------------------------------------------------------------------
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include "utils/injection_point.h"
+#include "funcapi.h"
+#include "utils/injection_point.h"
+#include "storage/dsm_registry.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
+
+/*
+ * memcontext_crash_client
+ *
+ * Ensure that the client process aborts in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_client);
+Datum
+memcontext_crash_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-client-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(memcontext_detach_client);
+Datum
+memcontext_detach_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointDetach("memcontext-client-crash");
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_crash_server
+ *
+ * Ensure that the server process crashes in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_server);
+Datum
+memcontext_crash_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-server-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_detach_server
+ *
+ * Detach the injection point which crashes the server
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_detach_server);
+Datum
+memcontext_detach_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointDetach("memcontext-server-crash");
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * dsa_dump_sql
+ */
+PG_FUNCTION_INFO_V1(dsa_dump_sql);
+Datum
+dsa_dump_sql(PG_FUNCTION_ARGS)
+{
+ bool found;
+ size_t tot_size;
+ dsa_area *memstats_dsa_area;
+
+ memstats_dsa_area = pg_get_memstats_dsa_area();
+
+ if (memstats_dsa_area == NULL)
+ memstats_dsa_area = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ tot_size = dsa_get_total_size(memstats_dsa_area);
+ dsa_detach(memstats_dsa_area);
+ PG_RETURN_INT64(tot_size);
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-11-20 09:56 Daniel Gustafsson <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Daniel Gustafsson @ 2025-11-20 09:56 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: pgsql-hackers; torikoshia <[email protected]>
> On 7 Nov 2025, at 23:55, Rahila Syed <[email protected]> wrote:
> I have attached a version 40 patch that has been rebased onto the
> latest master branch, as CFbot indicated a rebase was needed.
Thanks for the rebase, below are a few mostly superficial comments on the
patch:
+#include "access/twophase.h"
+#include "catalog/pg_authid_d.h"
...
+#include "utils/acl.h"
Are these actually required to be included?
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
Why is this needed? MemoryStatsContextId is identical to MemoryContextId and
is too only used in mcxtfuncs.c so there is no need to expose it in memutils.h.
Can't you just use MemoryContextId everywhere or am I missing something?
+#define CLIENT_KEY_SIZE 64
+
+static LWLock *client_keys_lock = NULL;
+static int *client_keys = NULL;
+static dshash_table *MemoryStatsDsHash = NULL;
+static dsa_area *MemoryStatsDsaArea = NULL;
These new additions have in some cases too generic names (client_keys etc) and
they all lack comments explaining why they're needed. Maybe a leading block
comment explaining they are used for process memory context reporting, and then
inline comments on each with their use?
+#define CLIENT_KEY_SIZE 64
...
+ char key[CLIENT_KEY_SIZE];
...
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
Given that MyProcNumber is an index into the proc array, it seems excessive to
use 64 bytes to store it, can't we get away with a small stack allocation?
+ * Retreive the client key for publishing statistics and reset it to -1,
s/Retreive/Retrieve/
+ ProcNumber procNumber = INVALID_PROC_NUMBER;
This variable is never accessed before getting re-assigned, so this assignment
in the variable definition can be removed per project style.
+ InitMaterializedSRF(fcinfo, 0);
Can this initialization be postponed till when we know the ResultSetInfo is
needed? While a micro optimization, it seems we can avoid that overhead in
case the query errors out?
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params, &found);
Nitpick, but there are a few oversize lines, like this one, which need to be
wrapped to match project style.
+ /*
+ * XXX. If the process exits without cleaning up its slot, i.e in case of
+ * an abrupt crash the client_keys slot won't be reset thus resulting in
+ * false negative and WARNING would be thrown in case another process with
+ * same slot index is queried for statistics.
+ */
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request", pid));
+ PG_RETURN_NULL();
+ }
AFAICT this mean that a failure to clean up (through a crash for example) can
block a future backend from reporting which isn't entirely ideal. Is there
anything we can do to mitigate this?
+ bool summary = false;
In ProcessGetMemoryContextInterrupt(), can't we just read entry->summary rather
than define a local variable and assign it? We already read lots of other
fields from entry directly so it seems more readable to be consistent.
+ /*
+ * Add the count of children contexts which are traversed
+ */
+ *num_contexts = *num_contexts + ichild;
Isn't this really the number of children + the parent context? ichild starts
at one to (AIUI) include the parent context. Also, MemoryContextStatsCounter
should also make sure to set num_contexts to zero before adding to it.
+#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
I don't think MAX_MEMORY_CONTEXT_STATS_SIZE adds any value as it's only used
once, on the line directly after its definition. We can just use the expansion
of ((sizeof(MemoryStatsEntry)) when defining MAX_MEMORY_CONTEXT_STATS_NUM.
> The test module patch is unchanged.
Please include all commits in the series even if they aren't updated since the
CFBot cannot pick them up otherwise.
--
Daniel Gustafsson
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-11-25 07:20 Rahila Syed <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-11-25 07:20 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: pgsql-hackers; torikoshia <[email protected]>
Hi Daniel,
Thank you for your comments. Please find attached v41 with all the comments
addressed.
>
> +#include "access/twophase.h"
> +#include "catalog/pg_authid_d.h"
> ...
> +#include "utils/acl.h"
> Are these actually required to be included?
>
>
Removed these.
> - MemoryContextId *entry;
> + MemoryStatsContextId *entry;
> Why is this needed? MemoryStatsContextId is identical to MemoryContextId
> and
> is too only used in mcxtfuncs.c so there is no need to expose it in
> memutils.h.
> Can't you just use MemoryContextId everywhere or am I missing something?
>
>
MemoryContextId has been renamed to MemoryStatsContextId for better
code readability. I removed the leftover MemoryContextId definition.
Also, I moved it out of memutils.h. Did the same with some other structures
and definitions which were only used in mcxtfuncs.c
> +#define CLIENT_KEY_SIZE 64
> +
> +static LWLock *client_keys_lock = NULL;
> +static int *client_keys = NULL;
> +static dshash_table *MemoryStatsDsHash = NULL;
> +static dsa_area *MemoryStatsDsaArea = NULL;
> These new additions have in some cases too generic names (client_keys etc)
> and
> they all lack comments explaining why they're needed. Maybe a leading
> block
> comment explaining they are used for process memory context reporting, and
> then
> inline comments on each with their use?
>
>
Added comments.
> +#define CLIENT_KEY_SIZE 64
> ...
> + char key[CLIENT_KEY_SIZE];
> ...
> + snprintf(key, sizeof(key), "%d", MyProcNumber);
> Given that MyProcNumber is an index into the proc array, it seems
> excessive to
> use 64 bytes to store it, can't we get away with a small stack allocation?
>
I agree. Defined it as 32 bytes as MyProcNumber is of size uint32. Kindly
let me know if you think it can be reduced further.
+ * Retreive the client key for publishing statistics and reset it to -1,
> s/Retreive/Retrieve/
>
Fixed.
> + ProcNumber procNumber = INVALID_PROC_NUMBER;
> This variable is never accessed before getting re-assigned, so this
> assignment
> in the variable definition can be removed per project style.
>
>
>
Fixed too.
> + InitMaterializedSRF(fcinfo, 0);
> Can this initialization be postponed till when we know the ResultSetInfo is
> needed? While a micro optimization, it seems we can avoid that overhead in
> case the query errors out?
>
>
Good point. Added this just before the result set is getting populated.
> + if (MemoryStatsDsHash == NULL)
> + MemoryStatsDsHash =
> GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params,
> &found);
> Nitpick, but there are a few oversize lines, like this one, which need to
> be
> wrapped to match project style.
>
>
I have edited this accordingly.
> + /*
> + * XXX. If the process exits without cleaning up its slot, i.e in case
> of
> + * an abrupt crash the client_keys slot won't be reset thus resulting
> in
> + * false negative and WARNING would be thrown in case another process
> with
> + * same slot index is queried for statistics.
> + */
> + if (client_keys[procNumber] == -1)
> + client_keys[procNumber] = MyProcNumber;
> + else
> + {
> + LWLockRelease(client_keys_lock);
> + ereport(WARNING,
> + errmsg("server process %d is processing previous request",
> pid));
> + PG_RETURN_NULL();
> + }
> AFAICT this mean that a failure to clean up (through a crash for example)
> can
> block a future backend from reporting which isn't entirely ideal. Is there
> anything we can do to mitigate this?
>
>
Yes, we can reset it when the client times out, as long as we verify that
the value corresponds
to our ProcNumber and not another client's request. Fixed accordingly.
>
> + bool summary = false;
> In ProcessGetMemoryContextInterrupt(), can't we just read entry->summary
> rather
> than define a local variable and assign it? We already read lots of other
> fields from entry directly so it seems more readable to be consistent.
>
>
Fixed.
>
> + /*
> + * Add the count of children contexts which are traversed
> + */
> + *num_contexts = *num_contexts + ichild;
> Isn't this really the number of children + the parent context? ichild
> starts
> at one to (AIUI) include the parent context. Also,
> MemoryContextStatsCounter
> should also make sure to set num_contexts to zero before adding to it.
>
>
Yes. Adjusted the comment to match this and set num_contexts to zero.
>
> +#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
> +#define MAX_MEMORY_CONTEXT_STATS_NUM
> MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
> I don't think MAX_MEMORY_CONTEXT_STATS_SIZE adds any value as it's only
> used
> once, on the line directly after its definition. We can just use the
> expansion
> of ((sizeof(MemoryStatsEntry)) when defining MAX_MEMORY_CONTEXT_STATS_NUM.
>
>
Fixed.
I've attached the test patch as is, I will clean it up and do further
improvements to it.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v41-0001-Add-function-to-report-memory-context-statistics.patch (58.6K, ../../CAH2L28ucG7zLjLRU4m_hCaGfo_UwUyw89OZR9Q8t8YMriyijgw@mail.gmail.com/3-v41-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From f37252238a63e4023a06beac1b6e3dd0887fffcf Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Mon, 24 Nov 2025 17:05:47 +0530
Subject: [PATCH] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 157 +++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1013 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 11 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 4 +-
25 files changed, 1285 insertions(+), 30 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 1b465bc8ba7..a5c66837241 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,130 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +426,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 95ad29a64b9..6fe67e950bf 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -692,6 +692,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 1c38488f2cb..561d88ebb4d 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e84e8663e96..5b3e08805bf 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index ba63b84dfc5..29454b8bf8a 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ce6b5299324..fdd385e492d 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -871,6 +871,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c4a888a081c..00f03b36ed8 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index b23d0c19360..a5ed58a18c5 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -140,6 +141,7 @@ CalculateShmemSize(void)
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
InjectionPointShmemInit();
AioShmemInit();
WaitLSNShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 1504fafe6d8..c5e69151756 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 7dd75a490aa..e726f40dfbb 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3539,6 +3539,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index c1ac71ff7f2..644d8d988e1 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -162,6 +162,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -404,6 +405,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..8e0e3116f1f 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -17,28 +17,138 @@
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/* Size of dshash key */
+#define CLIENT_KEY_SIZE 32
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * These are used for reporting memory context
+ * statistics of a process.
+ */
+
+/* Lock to control access to client_keys array */
+static LWLock *client_keys_lock = NULL;
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to dsa memory containing
+ * memory statistics and other meta data. There is one
+ * entry per client backend request, keyed by ProcNumber of
+ * the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static void memstats_client_key_reset(int ProcNumber);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
- * MemoryContextId
+ * MemoryStatsContextId
* Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * pg_get_backend_memory_contexts and the likes.
*/
-typedef struct MemoryContextId
+typedef struct MemoryStatsContextId
{
MemoryContext context;
int context_id;
-} MemoryContextId;
+} MemoryStatsContextId;
/*
* int_list_to_array
@@ -89,7 +199,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +253,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +268,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +314,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +341,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +349,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +430,837 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give up
+ * and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is a warning because we don't want to break loops.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request", pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m", pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ INJECTION_POINT("memcontext-client-crash", NULL);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ INJECTION_POINT("memcontext-client-crash", NULL);
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000), WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum, path_length, INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in
+ * per-process DSA pointers. These pointers are stored in a dshash table with
+ * key as requesting clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process
+ */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ INJECTION_POINT("memcontext-server-crash", NULL);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Entry has been deleted due to client process exit. Make sure that the
+ * client always deletes the entry after taking required lock or this
+ * function may end up writing to unallocated memory.
+ */
+ if (!found)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /*
+ * The client has timed out waiting for us to write statistics and is
+ * requesting statistics from some other process
+ */
+ if (entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &c,
+ HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id -
+ num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting backends and return */
+ ConditionVariableBroadcast(&entry->memcxt_cv);
+}
+
+/*
+ * Clean up before exit from ProcessGetMemoryContextInterrupt
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry,
+ MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+/* Used for testing purposes */
+dsa_area *
+pg_get_memstats_dsa_area(void)
+{
+ if (MemoryStatsDsaArea != NULL)
+ return MemoryStatsDsaArea;
+ else
+ return NULL;
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 98f9598cd78..202403ebc63 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -658,6 +658,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..31c4de9f0b4 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1008,6 +1008,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 1edb18958f7..5e532b6df21 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8617,6 +8617,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 9a7d733ddef..b76f24baed6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 5b0ce383408..613e769c84e 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -136,3 +136,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7bbe5a36959..70df562c6dd 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,7 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "utils/dsa.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,7 +48,6 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
-
/*
* Standard top-level memory contexts.
*
@@ -149,6 +148,7 @@ extern MemoryContext BumpContextCreate(MemoryContext parent,
Size minContextSize,
Size initBlockSize,
Size maxBlockSize);
+extern dsa_area *pg_get_memstats_dsa_area(void);
/*
* Recommended default alloc parameters, suitable for "ordinary" contexts
@@ -319,4 +319,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3b37fafa65b..21c65ad2d10 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..c9da4fc8c90 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 57a8f0366a5..9bd241af59d 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1684,9 +1684,11 @@ MemoryContextCallback
MemoryContextCallbackFunction
MemoryContextCounters
MemoryContextData
-MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] v2-0002-Test-module-to-test-memory-context-reporting-with-in (1).patch (9.5K, ../../CAH2L28ucG7zLjLRU4m_hCaGfo_UwUyw89OZR9Q8t8YMriyijgw@mail.gmail.com/4-v2-0002-Test-module-to-test-memory-context-reporting-with-in%20%281%29.patch)
download | inline diff:
From e21832c610574f48496c961a7318070b720568de Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 23 Oct 2025 18:01:36 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 32 +++++
.../t/001_memcontext_inj.pl | 58 +++++++++
.../test_memcontext_reporting--1.0.sql | 11 ++
.../test_memcontext_reporting.c | 123 ++++++++++++++++++
.../test_memcontext_reporting.control | 4 +
6 files changed, 229 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 902a7954101..a31a2578c18 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -31,6 +31,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..01a7baa0263
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,32 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+EXTENSION = test_memcontext_reporting
+DATA = test_memcontext_reporting--1.0.sql
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..69d8489eb37
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,58 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init(allows_streaming => 1);
+# max_connections need to be bumped in order to accommodate for pgbench clients
+# and log_statement is dialled down since it otherwise will generate enormous
+# amounts of logging. Page verification failures are still logged.
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION test_memcontext_reporting;');
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+# Attaching to a client process injection point that throws an error
+$node->safe_psql('postgres', "select injection_points_attach('memcontext-client-crash', 'error');");
+
+my $pid = $node->safe_psql('postgres', "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+print "PID";
+print $pid;
+
+#Client should have thrown error
+$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+like ( $psql_err, qr/error triggered for injection point memcontext-client-crash/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres', "select injection_points_detach('memcontext-client-crash');");
+my $topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres', "select injection_points_attach('memcontext-server-crash', 'error');");
+
+#Server should have thrown error
+$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres', "select injection_points_detach('memcontext-server-crash');");
+$topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
new file mode 100644
index 00000000000..181daf429d0
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
@@ -0,0 +1,11 @@
+CREATE FUNCTION memcontext_crash_server()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION memcontext_crash_client()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION dsa_dump_sql()
+RETURNS bigint
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..955155524c2
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,123 @@
+/*
+ * -------------------------------------------------------------------------
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include "utils/injection_point.h"
+#include "funcapi.h"
+#include "utils/injection_point.h"
+#include "storage/dsm_registry.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
+
+/*
+ * memcontext_crash_client
+ *
+ * Ensure that the client process aborts in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_client);
+Datum
+memcontext_crash_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-client-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(memcontext_detach_client);
+Datum
+memcontext_detach_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointDetach("memcontext-client-crash");
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_crash_server
+ *
+ * Ensure that the server process crashes in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_server);
+Datum
+memcontext_crash_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-server-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_detach_server
+ *
+ * Detach the injection point which crashes the server
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_detach_server);
+Datum
+memcontext_detach_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointDetach("memcontext-server-crash");
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * dsa_dump_sql
+ */
+PG_FUNCTION_INFO_V1(dsa_dump_sql);
+Datum
+dsa_dump_sql(PG_FUNCTION_ARGS)
+{
+ bool found;
+ size_t tot_size;
+ dsa_area *memstats_dsa_area;
+
+ memstats_dsa_area = pg_get_memstats_dsa_area();
+
+ if (memstats_dsa_area == NULL)
+ memstats_dsa_area = GetNamedDSA("memory_context_statistics_dsa", &found);
+
+ tot_size = dsa_get_total_size(memstats_dsa_area);
+ dsa_detach(memstats_dsa_area);
+ PG_RETURN_INT64(tot_size);
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-11-28 09:22 Rahila Syed <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 2 replies; 65+ messages in thread
From: Rahila Syed @ 2025-11-28 09:22 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: pgsql-hackers; torikoshia <[email protected]>
Hi,
I'm attaching the updated patches, which primarily include cleanup and have
been rebased
following the CFbot report.
Thank you,
Rahila Syed
On Tue, Nov 25, 2025 at 12:50 PM Rahila Syed <[email protected]> wrote:
> Hi Daniel,
>
> Thank you for your comments. Please find attached v41 with all the
> comments addressed.
>
>
>>
>> +#include "access/twophase.h"
>> +#include "catalog/pg_authid_d.h"
>> ...
>> +#include "utils/acl.h"
>> Are these actually required to be included?
>>
>>
> Removed these.
>
>
>> - MemoryContextId *entry;
>> + MemoryStatsContextId *entry;
>> Why is this needed? MemoryStatsContextId is identical to MemoryContextId
>> and
>> is too only used in mcxtfuncs.c so there is no need to expose it in
>> memutils.h.
>> Can't you just use MemoryContextId everywhere or am I missing something?
>>
>>
> MemoryContextId has been renamed to MemoryStatsContextId for better
> code readability. I removed the leftover MemoryContextId definition.
> Also, I moved it out of memutils.h. Did the same with some other structures
> and definitions which were only used in mcxtfuncs.c
>
>
>> +#define CLIENT_KEY_SIZE 64
>> +
>> +static LWLock *client_keys_lock = NULL;
>> +static int *client_keys = NULL;
>> +static dshash_table *MemoryStatsDsHash = NULL;
>> +static dsa_area *MemoryStatsDsaArea = NULL;
>> These new additions have in some cases too generic names (client_keys
>> etc) and
>> they all lack comments explaining why they're needed. Maybe a leading
>> block
>> comment explaining they are used for process memory context reporting,
>> and then
>> inline comments on each with their use?
>>
>>
> Added comments.
>
>
>> +#define CLIENT_KEY_SIZE 64
>> ...
>> + char key[CLIENT_KEY_SIZE];
>> ...
>> + snprintf(key, sizeof(key), "%d", MyProcNumber);
>> Given that MyProcNumber is an index into the proc array, it seems
>> excessive to
>> use 64 bytes to store it, can't we get away with a small stack allocation?
>>
>
> I agree. Defined it as 32 bytes as MyProcNumber is of size uint32. Kindly
> let me know if you think it can be reduced further.
>
>
> + * Retreive the client key for publishing statistics and reset it to
>> -1,
>> s/Retreive/Retrieve/
>>
>
> Fixed.
>
>
>> + ProcNumber procNumber = INVALID_PROC_NUMBER;
>> This variable is never accessed before getting re-assigned, so this
>> assignment
>> in the variable definition can be removed per project style.
>>
>>
>>
> Fixed too.
>
>
>> + InitMaterializedSRF(fcinfo, 0);
>> Can this initialization be postponed till when we know the ResultSetInfo
>> is
>> needed? While a micro optimization, it seems we can avoid that overhead
>> in
>> case the query errors out?
>>
>>
> Good point. Added this just before the result set is getting populated.
>
>
>> + if (MemoryStatsDsHash == NULL)
>> + MemoryStatsDsHash =
>> GetNamedDSHash("memory_context_statistics_dshash", &memctx_dsh_params,
>> &found);
>> Nitpick, but there are a few oversize lines, like this one, which need to
>> be
>> wrapped to match project style.
>>
>>
> I have edited this accordingly.
>
>
>> + /*
>> + * XXX. If the process exits without cleaning up its slot, i.e in
>> case of
>> + * an abrupt crash the client_keys slot won't be reset thus resulting
>> in
>> + * false negative and WARNING would be thrown in case another process
>> with
>> + * same slot index is queried for statistics.
>> + */
>> + if (client_keys[procNumber] == -1)
>> + client_keys[procNumber] = MyProcNumber;
>> + else
>> + {
>> + LWLockRelease(client_keys_lock);
>> + ereport(WARNING,
>> + errmsg("server process %d is processing previous
>> request", pid));
>> + PG_RETURN_NULL();
>> + }
>> AFAICT this mean that a failure to clean up (through a crash for example)
>> can
>> block a future backend from reporting which isn't entirely ideal. Is
>> there
>> anything we can do to mitigate this?
>>
>>
> Yes, we can reset it when the client times out, as long as we verify that
> the value corresponds
> to our ProcNumber and not another client's request. Fixed accordingly.
>
>
>>
>> + bool summary = false;
>> In ProcessGetMemoryContextInterrupt(), can't we just read entry->summary
>> rather
>> than define a local variable and assign it? We already read lots of other
>> fields from entry directly so it seems more readable to be consistent.
>>
>>
> Fixed.
>
>
>>
>> + /*
>> + * Add the count of children contexts which are traversed
>> + */
>> + *num_contexts = *num_contexts + ichild;
>> Isn't this really the number of children + the parent context? ichild
>> starts
>> at one to (AIUI) include the parent context. Also,
>> MemoryContextStatsCounter
>> should also make sure to set num_contexts to zero before adding to it.
>>
>>
> Yes. Adjusted the comment to match this and set num_contexts to zero.
>
>
>>
>> +#define MAX_MEMORY_CONTEXT_STATS_SIZE (sizeof(MemoryStatsEntry))
>> +#define MAX_MEMORY_CONTEXT_STATS_NUM
>> MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / MAX_MEMORY_CONTEXT_STATS_SIZE
>> I don't think MAX_MEMORY_CONTEXT_STATS_SIZE adds any value as it's only
>> used
>> once, on the line directly after its definition. We can just use the
>> expansion
>> of ((sizeof(MemoryStatsEntry)) when defining MAX_MEMORY_CONTEXT_STATS_NUM.
>>
>>
> Fixed.
>
> I've attached the test patch as is, I will clean it up and do further
> improvements to it.
>
> Thank you,
> Rahila Syed
>
Attachments:
[application/octet-stream] v42-0002-Test-module-to-test-memory-context-reporting-with-in.patch (9.0K, ../../CAH2L28sMK4qyo9xH9Jy7UTj1seZDDi4+ATjSSKTcrfcvk_V=ig@mail.gmail.com/3-v42-0002-Test-module-to-test-memory-context-reporting-with-in.patch)
download | inline diff:
From d11c41bf3055a3a8a4c109e49a87a49ffaeaaa88 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 32 ++++++
.../t/001_memcontext_inj.pl | 58 ++++++++++
.../test_memcontext_reporting--1.0.sql | 7 ++
.../test_memcontext_reporting.c | 102 ++++++++++++++++++
.../test_memcontext_reporting.control | 4 +
6 files changed, 204 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index d079b91b1a2..1ed0cdc66b3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -32,6 +32,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..01a7baa0263
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,32 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+EXTENSION = test_memcontext_reporting
+DATA = test_memcontext_reporting--1.0.sql
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..69d8489eb37
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,58 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init(allows_streaming => 1);
+# max_connections need to be bumped in order to accommodate for pgbench clients
+# and log_statement is dialled down since it otherwise will generate enormous
+# amounts of logging. Page verification failures are still logged.
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION test_memcontext_reporting;');
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+# Attaching to a client process injection point that throws an error
+$node->safe_psql('postgres', "select injection_points_attach('memcontext-client-crash', 'error');");
+
+my $pid = $node->safe_psql('postgres', "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+print "PID";
+print $pid;
+
+#Client should have thrown error
+$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+like ( $psql_err, qr/error triggered for injection point memcontext-client-crash/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres', "select injection_points_detach('memcontext-client-crash');");
+my $topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres', "select injection_points_attach('memcontext-server-crash', 'error');");
+
+#Server should have thrown error
+$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres', "select injection_points_detach('memcontext-server-crash');");
+$topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
new file mode 100644
index 00000000000..4f787cf789e
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
@@ -0,0 +1,7 @@
+CREATE FUNCTION memcontext_crash_server()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION memcontext_crash_client()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..774ae7df49d
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,102 @@
+/*
+ * -------------------------------------------------------------------------
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include "utils/injection_point.h"
+#include "funcapi.h"
+#include "utils/injection_point.h"
+#include "storage/dsm_registry.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
+
+/*
+ * memcontext_crash_client
+ *
+ * Ensure that the client process aborts in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_client);
+Datum
+memcontext_crash_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-client-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(memcontext_detach_client);
+Datum
+memcontext_detach_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointDetach("memcontext-client-crash");
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_crash_server
+ *
+ * Ensure that the server process crashes in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_server);
+Datum
+memcontext_crash_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-server-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_detach_server
+ *
+ * Detach the injection point which crashes the server
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_detach_server);
+Datum
+memcontext_detach_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointDetach("memcontext-server-crash");
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
[application/octet-stream] v42-0001-Add-function-to-report-memory-context-statistics.patch (58.2K, ../../CAH2L28sMK4qyo9xH9Jy7UTj1seZDDi4+ATjSSKTcrfcvk_V=ig@mail.gmail.com/4-v42-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From a929a519d994a4ced1fd69ce64dd83d5bfc8ff19 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 157 +++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1011 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 10 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 4 +-
25 files changed, 1282 insertions(+), 30 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 1b465bc8ba7..a5c66837241 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,130 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +426,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 6fffdb9398e..47c5422f4ad 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -692,6 +692,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 1c38488f2cb..561d88ebb4d 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e84e8663e96..5b3e08805bf 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index ba63b84dfc5..29454b8bf8a 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ce6b5299324..fdd385e492d 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -871,6 +871,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c4a888a081c..00f03b36ed8 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index b23d0c19360..a5ed58a18c5 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -140,6 +141,7 @@ CalculateShmemSize(void)
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
InjectionPointShmemInit();
AioShmemInit();
WaitLSNShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 1504fafe6d8..c5e69151756 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 7dd75a490aa..e726f40dfbb 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3539,6 +3539,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index c1ac71ff7f2..644d8d988e1 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -162,6 +162,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -404,6 +405,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..c661eef7ae9 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -17,28 +17,138 @@
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/* Size of dshash key */
+#define CLIENT_KEY_SIZE 32
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * These are used for reporting memory context
+ * statistics of a process.
+ */
+
+/* Lock to control access to client_keys array */
+static LWLock *client_keys_lock = NULL;
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to dsa memory containing
+ * memory statistics and other meta data. There is one
+ * entry per client backend request, keyed by ProcNumber of
+ * the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static void memstats_client_key_reset(int ProcNumber);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
- * MemoryContextId
+ * MemoryStatsContextId
* Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * pg_get_backend_memory_contexts and the likes.
*/
-typedef struct MemoryContextId
+typedef struct MemoryStatsContextId
{
MemoryContext context;
int context_id;
-} MemoryContextId;
+} MemoryStatsContextId;
/*
* int_list_to_array
@@ -89,7 +199,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +253,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +268,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +314,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +341,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +349,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +430,835 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give up
+ * and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is a warning because we don't want to break loops.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ INJECTION_POINT("memcontext-client-crash", NULL);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in
+ * per-process DSA pointers. These pointers are stored in a dshash table with
+ * key as requesting clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process
+ */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ INJECTION_POINT("memcontext-server-crash", NULL);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Entry has been deleted due to client process exit. Make sure that the
+ * client always deletes the entry after taking required lock or this
+ * function may end up writing to unallocated memory.
+ */
+ if (!found)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /*
+ * The client has timed out waiting for us to write statistics and is
+ * requesting statistics from some other process
+ */
+ if (entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id
+ - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+}
+
+/*
+ * Clean up before exit from ProcessGetMemoryContextInterrupt
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry,
+ MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 98f9598cd78..202403ebc63 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -658,6 +658,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..31c4de9f0b4 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1008,6 +1008,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 66431940700..8af5c016365 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8617,6 +8617,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 9a7d733ddef..b76f24baed6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 5b0ce383408..613e769c84e 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -136,3 +136,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7bbe5a36959..4296667cbf0 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,7 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "utils/dsa.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,7 +48,6 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
-
/*
* Standard top-level memory contexts.
*
@@ -319,4 +318,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3b37fafa65b..21c65ad2d10 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..c9da4fc8c90 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index e3c3523b5b2..d9b45bdd721 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1686,9 +1686,11 @@ MemoryContextCallback
MemoryContextCallbackFunction
MemoryContextCounters
MemoryContextData
-MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-12-08 01:26 torikoshia <[email protected]>
parent: Rahila Syed <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: torikoshia @ 2025-12-08 01:26 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; pgsql-hackers
On 2025-11-28 18:22, Rahila Syed wrote:
Hi,
> I'm attaching the updated patches, which primarily include cleanup and
> have been rebased
> following the CFbot report.
Thanks for updating the patch!
I observed an assertion failure when forcing a timeout as follows:
```
$ psql
(pid:38587)=#
$ kill -s SIGSTOP 38587
$ psql
(pid:38618) =# select * from pg_get_process_memory_contexts(38587,
false);
name | ident | type | level | path | total_bytes |
total_nblocks | free_bytes | free_chunks | used_bytes | num_agg_contexts
--------+--------+--------+--------+--------+-------------+---------------+------------+-------------+------------+------------------
[NULL] | [NULL] | [NULL] | [NULL] | [NULL] | [NULL] |
[NULL] | [NULL] | [NULL] | [NULL] | [NULL]
(1 row)
Time: 5013.515 ms (00:05.014)
$ kill -s SIGCONT 38587
$ tail postgresql.log
TRAP: failed Assert("client_keys[MyProcNumber] != -1"), File:
"mcxtfuncs.c", Line: 881, PID: 38587
0 postgres 0x0000000104943400
ExceptionalCondition + 216
1 postgres 0x000000010480f738
ProcessGetMemoryContextInterrupt + 140
2 postgres 0x00000001046f2710
ProcessInterrupts + 3008
3 postgres 0x00000001046f1a78
ProcessClientReadInterrupt + 80
4 postgres 0x0000000104433994 secure_read
+ 404
5 postgres 0x00000001044411dc pq_recvbuf
+ 260
6 postgres 0x0000000104441088 pq_getbyte
+ 96
7 postgres 0x00000001046fa0fc
SocketBackend + 44
8 postgres 0x00000001046f6d3c ReadCommand
+ 44
9 postgres 0x00000001046f6284
PostgresMain + 2900
10 postgres 0x00000001046ed558
BackendInitialize + 0
11 postgres 0x00000001045c0a48
postmaster_child_launch + 456
12 postgres 0x00000001045c8520
BackendStartup + 304
13 postgres 0x00000001045c636c ServerLoop
+ 372
14 postgres 0x00000001045c4e24
PostmasterMain + 6448
15 postgres 0x0000000104445b4c main + 924
16 dyld 0x0000000188662b98 start +
6076
2025-12-08 07:35:32.608 JST [38540] LOG: 00000: client backend (PID
38587) was terminated by signal 6: Abort trap: 6
2025-12-08 07:35:32.608 JST [38540] LOCATION: LogChildExit,
postmaster.c:2872
```
Below are comments regarding the v42-0001 patch:
> In order to not block on busy processes, we have hardcoded
> the number of seconds during which to retry before timing out.
> In the case where no statistics are published within the set
> timeout, NULL is returned.
It might be good to also document in func-admin.sgml that the function
times out after 5 seconds when the target backend does not respond, and
that in such a case NULLs are returned.
+ * If DSA exists, created by another process requesting statistics,
attach
+ * to it. We expect the client process to create required DSA and
Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea =
GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash =
GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
From the comment, it sounded to me as if the client executing
pg_get_process_memory_contexts() might not create the DSA in some cases.
Is it correct to assume that such a situation can happen?
In [1], as a response to concerns about using DSA inside a CFI handler,
you wrote that “all the dynamic shared memory needed to store the
statistics is created and deleted in the client function”.
So I understood that it would never create the DSA inside the CFI
handler.
If that understanding is correct, perhaps the comment should be reworded
to make that clear.
+ context_id_lookup =
hash_create("pg_get_remote_backend_memory_contexts",
This appears to use the old function name. Should this be updated to
"pg_get_process_memory_contexts" instead?
[1]
https://www.postgresql.org/message-id/CAH2L28sc-rEhyntPLoaC2XUa0ZjS5ka6KzEbuSVxQBBnUYu1KQ%40mail.gma...
--
Regards,
--
Atsushi Torikoshi
Seconded from NTT DATA Japan Corporation to SRA OSS K.K.
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-12-08 09:43 Daniel Gustafsson <[email protected]>
parent: Rahila Syed <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: Daniel Gustafsson @ 2025-12-08 09:43 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: pgsql-hackers; torikoshia <[email protected]>
> On 28 Nov 2025, at 10:22, Rahila Syed <[email protected]> wrote:
>
> Hi,
>
> I'm attaching the updated patches, which primarily include cleanup and have been rebased
> following the CFbot report.
Thanks for the patch, below are a few comments and suggestions. As I was
reviewing I tweaked the below and have attached the comments as changes in
0003.
== in func-admin.sgml
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type> )
We recently simplified the UI by removing the timeout, and the more I think
about it the more I am convinced that there is more simplification to be had.
The most likely usage pattern, IMO, will be to get all the contexts and not the
summary, so we can make the summary parameter DEFAULT to false. This allows
most uses to just pass the pid, without complicating the code at all.
== in mcxtfuncs.c
+/* Size of dshash key */
+#define CLIENT_KEY_SIZE 32
That's still pretty generous isn't it? We are printing a uint32 into it so the
highest number it can reach is ~4 billion (which while the upper limit, is
quite theoretic in this case). 10 + 1 bytes should suffice to store right?
- * MemoryContextId
+ * MemoryStatsContextId
Sorry, but I still don't agree with this rename and I think we should skip it,
if only to avoid changes to existing parts of the code.
+ * Entry has been deleted due to client process exit. Make sure that the
+ * client always deletes the entry after taking required lock or this
+ * function may end up writing to unallocated memory.
Can you explain this a bit further, I'm not sure I get it. The code goes on to
release a lock immediately and then destroys the hash. Who is responsible for
destroying the entry?
== In system-views.sql
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
This is not a view, and the functions aren't used to drive a view, so these
should not be defined here. The above mentioned change to add DEFAULT handling
to the summary parameter fixes this in the attached.
== In ProcessGetMemoryContextInterrupt()
I'm not a fan of having to exit's from the function doing duplicative cleanups,
in the attached I've wrapped them in a conditional to just have one exit path.
What do you think about that?
== In PublishMemoryContext()
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
Looking at this more I don't really like that resetting the memory context is
done via a separate function, when that function must be called from the exact
right place to ensure CurrentMemoryContext is what it thinks it is. It's all a
bit too magic. Since this is only called in 3 places I would prefer to inline
the code in PublishMemoryContext().
== In PublishMemoryContext()
+ const char *ident = context->ident;
+ const char *name = context->name;
ident and name are defined as const, but they are later assigned to after the
initial assignment. I think we need to unconstify these.
== In PublishMemoryContext()
+ if (strlen(name) >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
We already have namelen which is set to exactly strlen(name), so let's reuse
that for readability.
== In PublishMemoryContext()
+ /* Allocate DSA memory for storing path information */
This comment is no longer accurate is it? The DSA has already been allocated
at this point.
== In memutils.h
+#include "utils/dsa.h"
This is not needed.
I also did some smaller comment rewording and reflowing, some smaller cleanups
and a fresh pgindent/pgperltidy run. The attached 0003 contains the above.
--
Daniel Gustafsson
Attachments:
[application/octet-stream] v43-0001-Add-function-to-report-memory-context-statistics.patch (58.2K, ../../[email protected]/2-v43-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From 8f50ab607db07dc125fda2831e1d234c2a630ccb Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH v43 1/3] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 157 +++
src/backend/catalog/system_views.sql | 5 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1011 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 10 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 4 +-
25 files changed, 1282 insertions(+), 30 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 1b465bc8ba7..a5c66837241 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,130 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal>,
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +426,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 086c4c8fb6f..17ec512622b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -692,6 +692,11 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
+REVOKE EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION
+ pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 1c38488f2cb..561d88ebb4d 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e84e8663e96..5b3e08805bf 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index ba63b84dfc5..29454b8bf8a 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index ce6b5299324..fdd385e492d 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -871,6 +871,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c4a888a081c..00f03b36ed8 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index b23d0c19360..a5ed58a18c5 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -140,6 +141,7 @@ CalculateShmemSize(void)
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
InjectionPointShmemInit();
AioShmemInit();
WaitLSNShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index ebc3f4ca457..27b3b51cf2d 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 7dd75a490aa..e726f40dfbb 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3539,6 +3539,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index c1ac71ff7f2..644d8d988e1 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -162,6 +162,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -404,6 +405,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index fe6dce9cba3..c661eef7ae9 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -17,28 +17,138 @@
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/* Size of dshash key */
+#define CLIENT_KEY_SIZE 32
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * These are used for reporting memory context
+ * statistics of a process.
+ */
+
+/* Lock to control access to client_keys array */
+static LWLock *client_keys_lock = NULL;
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to dsa memory containing
+ * memory statistics and other meta data. There is one
+ * entry per client backend request, keyed by ProcNumber of
+ * the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static void memstats_client_key_reset(int ProcNumber);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
+static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
+ HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
- * MemoryContextId
+ * MemoryStatsContextId
* Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * pg_get_backend_memory_contexts and the likes.
*/
-typedef struct MemoryContextId
+typedef struct MemoryStatsContextId
{
MemoryContext context;
int context_id;
-} MemoryContextId;
+} MemoryStatsContextId;
/*
* int_list_to_array
@@ -89,7 +199,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -143,24 +253,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +268,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -189,7 +314,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryContextId);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -216,7 +341,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryContextId *entry;
+ MemoryStatsContextId *entry;
bool found;
/*
@@ -224,8 +349,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -305,3 +430,835 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. This is because allowing
+ * any users to issue this request at an unbounded rate would cause lots of
+ * requests to be sent, which can lead to denial of service. Additional roles
+ * can be permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give up
+ * and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ /*
+ * This is a warning because we don't want to break loops.
+ */
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send dsa pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The dsa pointers containing statistics for each client are stored in a
+ * dshash table. In addition to dsa pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ INJECTION_POINT("memcontext-client-crash", NULL);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in
+ * per-process DSA pointers. These pointers are stored in a dshash table with
+ * key as requesting clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts
+ * statistics. For that we traverse the memory context tree recursively in
+ * depth first search manner to cover all the children of a parent context, to
+ * be able to display a cumulative total of memory consumption by a parent at
+ * level 2 and all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process
+ */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ Assert(client_keys[MyProcNumber] != -1);
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL, "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * If DSA exists, created by another process requesting statistics, attach
+ * to it. We expect the client process to create required DSA and Dshash
+ * table.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ INJECTION_POINT("memcontext-server-crash", NULL);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Entry has been deleted due to client process exit. Make sure that the
+ * client always deletes the entry after taking required lock or this
+ * function may end up writing to unallocated memory.
+ */
+ if (!found)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /*
+ * The client has timed out waiting for us to write statistics and is
+ * requesting statistics from some other process
+ */
+ if (entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+ return;
+ }
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryStatsContextId *contextid_entry;
+
+ contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of its
+ * ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id is
+ * greater than max_stats, we stop reporting individual statistics
+ * when context_id equals max_stats - 2. As we use max_stats - 1 array
+ * slot for reporting cumulative statistics or "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Statistics are not aggregated, i.e individual statistics reported when
+ * context_id <= max_stats.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+ /* Report number of aggregated memory contexts */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id
+ - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for cumulative
+ * statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ entry->stats_written = true;
+ end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+}
+
+/*
+ * Clean up before exit from ProcessGetMemoryContextInterrupt
+ */
+static void
+end_memorycontext_reporting(MemoryStatsDSHashEntry *entry,
+ MemoryContext oldcontext, HTAB *context_id_lookup)
+{
+ MemoryContext curr_ctx = CurrentMemoryContext;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(curr_ctx);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryStatsContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ const char *ident = context->ident;
+ const char *name = context->name;
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = context->ident;
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (strlen(name) >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Allocate DSA memory for storing path information */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 98f9598cd78..202403ebc63 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -658,6 +658,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..31c4de9f0b4 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1008,6 +1008,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 66af2d96d67..5713bbb1550 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8617,6 +8617,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 9a7d733ddef..b76f24baed6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 5b0ce383408..613e769c84e 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -136,3 +136,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7bbe5a36959..4296667cbf0 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,7 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-
+#include "utils/dsa.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,7 +48,6 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
-
/*
* Standard top-level memory contexts.
*
@@ -319,4 +318,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3b37fafa65b..21c65ad2d10 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..c9da4fc8c90 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index cf3f6a7dafd..aa898f025c0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1686,9 +1686,11 @@ MemoryContextCallback
MemoryContextCallbackFunction
MemoryContextCounters
MemoryContextData
-MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsContextId
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.39.3 (Apple Git-146)
[application/octet-stream] v43-0002-Test-module-to-test-memory-context-reporting-wit.patch (9.0K, ../../[email protected]/3-v43-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From 00c38517db21893543f0af5b0c2f4eccc0a23658 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH v43 2/3] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 32 ++++++
.../t/001_memcontext_inj.pl | 58 ++++++++++
.../test_memcontext_reporting--1.0.sql | 7 ++
.../test_memcontext_reporting.c | 102 ++++++++++++++++++
.../test_memcontext_reporting.control | 4 +
6 files changed, 204 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index d079b91b1a2..1ed0cdc66b3 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -32,6 +32,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..01a7baa0263
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,32 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+EXTENSION = test_memcontext_reporting
+DATA = test_memcontext_reporting--1.0.sql
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..69d8489eb37
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,58 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init(allows_streaming => 1);
+# max_connections need to be bumped in order to accommodate for pgbench clients
+# and log_statement is dialled down since it otherwise will generate enormous
+# amounts of logging. Page verification failures are still logged.
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION test_memcontext_reporting;');
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+# Attaching to a client process injection point that throws an error
+$node->safe_psql('postgres', "select injection_points_attach('memcontext-client-crash', 'error');");
+
+my $pid = $node->safe_psql('postgres', "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+print "PID";
+print $pid;
+
+#Client should have thrown error
+$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+like ( $psql_err, qr/error triggered for injection point memcontext-client-crash/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres', "select injection_points_detach('memcontext-client-crash');");
+my $topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres', "select injection_points_attach('memcontext-server-crash', 'error');");
+
+#Server should have thrown error
+$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres', "select injection_points_detach('memcontext-server-crash');");
+$topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+ok($topcontext_name = 'TopMemoryContext');
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
new file mode 100644
index 00000000000..4f787cf789e
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting--1.0.sql
@@ -0,0 +1,7 @@
+CREATE FUNCTION memcontext_crash_server()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE FUNCTION memcontext_crash_client()
+RETURNS pg_catalog.void
+AS 'MODULE_PATHNAME' LANGUAGE C;
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..774ae7df49d
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,102 @@
+/*
+ * -------------------------------------------------------------------------
+ *
+ * Copyright (c) 2025, PostgreSQL Global Development Group
+ *
+ * IDENTIFICATION
+ * src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
+ *
+ * -------------------------------------------------------------------------
+ */
+
+#include "postgres.h"
+#include "utils/injection_point.h"
+#include "funcapi.h"
+#include "utils/injection_point.h"
+#include "storage/dsm_registry.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
+
+/*
+ * memcontext_crash_client
+ *
+ * Ensure that the client process aborts in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_client);
+Datum
+memcontext_crash_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-client-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+PG_FUNCTION_INFO_V1(memcontext_detach_client);
+Datum
+memcontext_detach_client(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointDetach("memcontext-client-crash");
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_crash_server
+ *
+ * Ensure that the server process crashes in between memory context
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_crash_server);
+Datum
+memcontext_crash_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointAttach("memcontext-server-crash",
+ "test_memcontext_reporting", "crash", NULL, 0);
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
+
+/*
+ * memcontext_detach_server
+ *
+ * Detach the injection point which crashes the server
+ * reporting.
+ */
+PG_FUNCTION_INFO_V1(memcontext_detach_server);
+Datum
+memcontext_detach_server(PG_FUNCTION_ARGS)
+{
+#ifdef USE_INJECTION_POINTS
+ InjectionPointDetach("memcontext-server-crash");
+
+#else
+ elog(ERROR,
+ "test is not working as intended when injection points are disabled");
+#endif
+ PG_RETURN_VOID();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.39.3 (Apple Git-146)
[application/octet-stream] v43-0003-Review-comments.patch (29.1K, ../../[email protected]/4-v43-0003-Review-comments.patch)
download | inline diff:
From 257b7d87aaa55c415e205afad9ba6ae90e8d6195 Mon Sep 17 00:00:00 2001
From: Daniel Gustafsson <[email protected]>
Date: Mon, 8 Dec 2025 10:38:43 +0100
Subject: [PATCH v43 3/3] Review comments
---
doc/src/sgml/func/func-admin.sgml | 4 +-
src/backend/catalog/system_functions.sql | 14 +
src/backend/catalog/system_views.sql | 5 -
src/backend/utils/adt/mcxtfuncs.c | 314 +++++++++---------
src/include/utils/memutils.h | 2 +-
.../t/001_memcontext_inj.pl | 42 ++-
src/tools/pgindent/typedefs.list | 2 +-
7 files changed, 195 insertions(+), 188 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index a5c66837241..3e71ced60a2 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -257,7 +257,7 @@
<indexterm>
<primary>pg_get_process_memory_contexts</primary>
</indexterm>
- <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type>, <parameter>summary</parameter> <type>boolean</type> )
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
<returnvalue>setof record</returnvalue>
( <parameter>name</parameter> <type>text</type>,
<parameter>ident</parameter> <type>text</type>,
@@ -360,7 +360,7 @@
Statistics for contexts on level 2 and below are aggregates of all
child contexts' statistics, where <literal>num_agg_contexts</literal>
indicate the number aggregated child contexts. When
- <parameter>summary</parameter> is <literal>false</literal>,
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
<literal>the num_agg_contexts</literal> value is <literal>1</literal>,
indicating that individual statistics are being displayed.
</para>
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 2d946d6d9e9..7b40bac5f57 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -657,6 +657,17 @@ LANGUAGE INTERNAL
STRICT VOLATILE PARALLEL UNSAFE
AS 'pg_replication_origin_session_setup';
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -782,6 +793,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -808,6 +820,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 17ec512622b..086c4c8fb6f 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -692,11 +692,6 @@ GRANT SELECT ON pg_backend_memory_contexts TO pg_read_all_stats;
REVOKE EXECUTE ON FUNCTION pg_get_backend_memory_contexts() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_get_backend_memory_contexts() TO pg_read_all_stats;
-REVOKE EXECUTE ON FUNCTION
- pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
-GRANT EXECUTE ON FUNCTION
- pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
-
-- Statistics views
CREATE VIEW pg_stat_all_tables AS
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c661eef7ae9..f1b5c3a0887 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -49,8 +49,11 @@
*/
#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
-/* Size of dshash key */
-#define CLIENT_KEY_SIZE 32
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
/* Dynamic shared memory state for reporting statistics per context */
typedef struct MemoryStatsEntry
@@ -92,8 +95,7 @@ static const dshash_parameters memctx_dsh_params = {
};
/*
- * These are used for reporting memory context
- * statistics of a process.
+ * These are used for reporting memory context statistics of a process.
*/
/* Lock to control access to client_keys array */
@@ -103,10 +105,9 @@ static LWLock *client_keys_lock = NULL;
static int *client_keys = NULL;
/*
- * Table to store pointers to dsa memory containing
- * memory statistics and other meta data. There is one
- * entry per client backend request, keyed by ProcNumber of
- * the client obtained from client_keys array above.
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
*/
static dshash_table *MemoryStatsDsHash = NULL;
@@ -125,8 +126,6 @@ static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
MemoryContextCounters stat,
int num_contexts);
static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
-static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryContext oldcontext,
- HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
@@ -140,15 +139,14 @@ static void end_memorycontext_reporting(MemoryStatsDSHashEntry *entry, MemoryCon
#define MEMORY_STATS_MAX_TIMEOUT 5
/*
- * MemoryStatsContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts and the likes.
+ * MemoryContextId
+ * Used for storage of transient identifiers for memory context reporting
*/
-typedef struct MemoryStatsContextId
+typedef struct MemoryContextId
{
MemoryContext context;
int context_id;
-} MemoryStatsContextId;
+} MemoryContextId;
/*
* int_list_to_array
@@ -199,7 +197,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
*/
for (MemoryContext cur = context; cur != NULL; cur = cur->parent)
{
- MemoryStatsContextId *entry;
+ MemoryContextId *entry;
bool found;
entry = hash_search(context_id_lookup, &cur, HASH_FIND, &found);
@@ -314,7 +312,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
HTAB *context_id_lookup;
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.entrysize = sizeof(MemoryContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_backend_memory_contexts",
@@ -341,7 +339,7 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
foreach_ptr(MemoryContextData, cur, contexts)
{
- MemoryStatsContextId *entry;
+ MemoryContextId *entry;
bool found;
/*
@@ -349,8 +347,8 @@ pg_get_backend_memory_contexts(PG_FUNCTION_ARGS)
* PutMemoryContextsStatsTupleStore needs this to populate the "path"
* column with the parent context_ids.
*/
- entry = (MemoryStatsContextId *) hash_search(context_id_lookup, &cur,
- HASH_ENTER, &found);
+ entry = (MemoryContextId *) hash_search(context_id_lookup, &cur,
+ HASH_ENTER, &found);
entry->context_id = context_id++;
Assert(!found);
@@ -437,10 +435,8 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
* wait for the results and display them.
*
* By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
- * to signal a process to return the memory contexts. This is because allowing
- * any users to issue this request at an unbounded rate would cause lots of
- * requests to be sent, which can lead to denial of service. Additional roles
- * can be permitted with GRANT.
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
*
* On receipt of this signal, a backend or an auxiliary process sets the flag
* in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
@@ -453,14 +449,14 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
* at the end of the buffer.
*
* After sending the signal, wait on a condition variable. The publishing
- * backend, after copying the data to shared memory, sends signal on that
+ * backend, after copying the data to shared memory, sends a signal on that
* condition variable. There is one condition variable per client process.
* Once the condition variable is signalled, check if the latest memory context
* information is available and display.
*
* If the publishing backend does not respond before the condition variable
- * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give up
- * and return NULL.
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
*/
Datum
pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
@@ -495,12 +491,8 @@ pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
*/
if (proc == NULL)
{
- /*
- * This is a warning because we don't want to break loops.
- */
ereport(WARNING,
- errmsg("PID %d is not a PostgreSQL server process",
- pid));
+ errmsg("PID %d is not a PostgreSQL server process", pid));
PG_RETURN_NULL();
}
@@ -508,7 +500,7 @@ pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
/*
* Create a DSA to allocate memory for copying memory contexts statistics.
- * Allocate the memory in the DSA and send dsa pointer to the server
+ * Allocate the memory in the DSA and send DSA pointer to the server
* process for storing the context statistics. If number of contexts
* exceed a predefined limit (1MB), a cumulative total is stored for such
* contexts.
@@ -521,8 +513,8 @@ pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
&found);
/*
- * The dsa pointers containing statistics for each client are stored in a
- * dshash table. In addition to dsa pointer, each entry in this table also
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
* contains information about the statistics, condition variable for
* signalling between client and the server and miscellaneous data
* specific to a request. There is one entry per client request in the
@@ -838,9 +830,9 @@ HandleGetMemoryContextInterrupt(void)
* output.
*
* Statistics for all the processes are shared via the same dynamic shared
- * area. Individual statistics are tracked independently in
- * per-process DSA pointers. These pointers are stored in a dshash table with
- * key as requesting clients ProcNumber.
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
*
* We calculate maximum number of context's statistics that can be displayed
* using a pre-determined limit for memory available per process for this
@@ -848,11 +840,11 @@ HandleGetMemoryContextInterrupt(void)
* context statistics if any are captured as a cumulative total at the end of
* individual context's statistics.
*
- * If summary is true, we capture the level 1 and level 2 contexts
- * statistics. For that we traverse the memory context tree recursively in
- * depth first search manner to cover all the children of a parent context, to
- * be able to display a cumulative total of memory consumption by a parent at
- * level 2 and all its children.
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
*/
void
ProcessGetMemoryContextInterrupt(void)
@@ -899,7 +891,7 @@ ProcessGetMemoryContextInterrupt(void)
* similar to its local backend counterpart.
*/
ctl.keysize = sizeof(MemoryContext);
- ctl.entrysize = sizeof(MemoryStatsContextId);
+ ctl.entrysize = sizeof(MemoryContextId);
ctl.hcxt = CurrentMemoryContext;
context_id_lookup = hash_create("pg_get_remote_backend_memory_contexts",
@@ -912,7 +904,7 @@ ProcessGetMemoryContextInterrupt(void)
/*
* If DSA exists, created by another process requesting statistics, attach
- * to it. We expect the client process to create required DSA and Dshash
+ * to it. We expect the client process to create required DSA and DSHash
* table.
*/
if (MemoryStatsDsaArea == NULL)
@@ -923,7 +915,6 @@ ProcessGetMemoryContextInterrupt(void)
MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
&memctx_dsh_params, &found);
-
snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
/*
@@ -935,25 +926,24 @@ ProcessGetMemoryContextInterrupt(void)
entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
/*
- * Entry has been deleted due to client process exit. Make sure that the
- * client always deletes the entry after taking required lock or this
- * function may end up writing to unallocated memory.
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * XXX ?: Make sure that the client always deletes the entry after taking
+ * required lock or this function may end up writing to unallocated
+ * memory.
*/
- if (!found)
+ if (!found || entry->target_server_id != MyProcPid)
{
entry->stats_written = false;
- end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
- return;
- }
- /*
- * The client has timed out waiting for us to write statistics and is
- * requesting statistics from some other process
- */
- if (entry->target_server_id != MyProcPid)
- {
- entry->stats_written = false;
- end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+
return;
}
@@ -966,7 +956,7 @@ ProcessGetMemoryContextInterrupt(void)
{
int cxt_id = 0;
List *path = NIL;
- MemoryStatsContextId *contextid_entry;
+ MemoryContextId *contextid_entry;
/* Copy TopMemoryContext statistics to DSA */
memset(&stat, 0, sizeof(stat));
@@ -976,9 +966,9 @@ ProcessGetMemoryContextInterrupt(void)
PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
1);
- contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup,
- &TopMemoryContext,
- HASH_ENTER, &found);
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
Assert(!found);
/*
@@ -1001,8 +991,8 @@ ProcessGetMemoryContextInterrupt(void)
memset(&grand_totals, 0, sizeof(grand_totals));
cxt_id++;
- contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup,
- &c, HASH_ENTER, &found);
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
Assert(!found);
contextid_entry->context_id = cxt_id + 1;
@@ -1014,119 +1004,111 @@ ProcessGetMemoryContextInterrupt(void)
grand_totals, num_contexts);
}
entry->total_stats = cxt_id + 1;
-
- entry->stats_written = true;
- end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
- /* Notify waiting client backend and return */
- ConditionVariableSignal(&entry->memcxt_cv);
- return;
}
- foreach_ptr(MemoryContextData, cur, contexts)
+ else
{
- List *path = NIL;
- MemoryStatsContextId *contextid_entry;
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
- contextid_entry = (MemoryStatsContextId *) hash_search(context_id_lookup,
- &cur,
- HASH_ENTER, &found);
- Assert(!found);
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
- /*
- * context id starts with 1
- */
- contextid_entry->context_id = context_id + 1;
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
/*
- * Figure out the transient context_id of this context and each of its
- * ancestors, to compute a path for this context.
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
*/
- path = compute_context_path(cur, context_id_lookup);
-
- /* Examine the context stats */
- memset(&stat, 0, sizeof(stat));
- (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
-
- /* Account for saving one statistics slot for cumulative reporting */
- if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
{
- /* Copy statistics to DSA memory */
- PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
- }
- else
- {
- meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
- meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
- meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
- meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
}
/*
- * DSA max limit per process is reached, write aggregate of the
- * remaining statistics.
- *
- * We can store contexts from 0 to max_stats - 1. When context_id is
- * greater than max_stats, we stop reporting individual statistics
- * when context_id equals max_stats - 2. As we use max_stats - 1 array
- * slot for reporting cumulative statistics or "Remaining Totals".
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
*/
- if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ else
{
- int namelen = strlen("Remaining Totals");
-
- num_individual_stats = context_id + 1;
- strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
- "Remaining Totals", namelen + 1);
- meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
- meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
- meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
}
- context_id++;
-
- for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
- contexts = lappend(contexts, c);
}
- /*
- * Statistics are not aggregated, i.e individual statistics reported when
- * context_id <= max_stats.
- */
- if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
- {
- entry->total_stats = context_id;
- meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
- }
- /* Report number of aggregated memory contexts */
- else
- {
- meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = context_id
- - num_individual_stats;
-
- /*
- * Total stats equals num_individual_stats + 1 record for cumulative
- * statistics.
- */
- entry->total_stats = num_individual_stats + 1;
- }
entry->stats_written = true;
- end_memorycontext_reporting(entry, oldcontext, context_id_lookup);
- /* Notify waiting client backend and return */
- ConditionVariableSignal(&entry->memcxt_cv);
-}
-
-/*
- * Clean up before exit from ProcessGetMemoryContextInterrupt
- */
-static void
-end_memorycontext_reporting(MemoryStatsDSHashEntry *entry,
- MemoryContext oldcontext, HTAB *context_id_lookup)
-{
- MemoryContext curr_ctx = CurrentMemoryContext;
-
dshash_release_lock(MemoryStatsDsHash, entry);
-
hash_destroy(context_id_lookup);
+
MemoryContextSwitchTo(oldcontext);
- MemoryContextReset(curr_ctx);
+ MemoryContextReset(memstats_ctx);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
}
/*
@@ -1144,7 +1126,7 @@ compute_context_path(MemoryContext c, HTAB *context_id_lookup)
for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
{
- MemoryStatsContextId *cur_entry;
+ MemoryContextId *cur_entry;
cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
@@ -1167,8 +1149,8 @@ PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
MemoryContext context, List *path,
MemoryContextCounters stat, int num_contexts)
{
- const char *ident = context->ident;
- const char *name = context->name;
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
/*
* To be consistent with logging output, we label dynahash contexts with
@@ -1176,7 +1158,7 @@ PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
*/
if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
{
- name = context->ident;
+ name = unconstify(char *, context->ident);
ident = NULL;
}
@@ -1184,7 +1166,7 @@ PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
{
int namelen = strlen(name);
- if (strlen(name) >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
namelen = pg_mbcliplen(name, namelen,
MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
@@ -1212,7 +1194,7 @@ PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
else
memcxt_info[curr_id].ident[0] = '\0';
- /* Allocate DSA memory for storing path information */
+ /* Store the path */
if (path == NIL)
memcxt_info[curr_id].path[0] = 0;
else
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 4296667cbf0..617de0ebf91 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -18,7 +18,6 @@
#define MEMUTILS_H
#include "nodes/memnodes.h"
-#include "utils/dsa.h"
/*
* MaxAllocSize, MaxAllocHugeSize
@@ -48,6 +47,7 @@
#define AllocHugeSizeIsValid(size) ((Size) (size) <= MaxAllocHugeSize)
+
/*
* Standard top-level memory contexts.
*
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
index 69d8489eb37..8fa12d1f693 100644
--- a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -11,7 +11,7 @@ use Test::More;
if ($ENV{enable_injection_points} ne 'yes')
{
- plan skip_all => 'Injection points not supported by this build';
+ plan skip_all => 'Injection points not supported by this build';
}
my $psql_err;
# Create and start a cluster with one node
@@ -21,8 +21,8 @@ $node->init(allows_streaming => 1);
# and log_statement is dialled down since it otherwise will generate enormous
# amounts of logging. Page verification failures are still logged.
$node->append_conf(
- 'postgresql.conf',
- qq[
+ 'postgresql.conf',
+ qq[
max_connections = 100
log_statement = none
]);
@@ -30,29 +30,45 @@ $node->start;
$node->safe_psql('postgres', 'CREATE EXTENSION test_memcontext_reporting;');
$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
# Attaching to a client process injection point that throws an error
-$node->safe_psql('postgres', "select injection_points_attach('memcontext-client-crash', 'error');");
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-crash', 'error');");
-my $pid = $node->safe_psql('postgres', "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
print "PID";
print $pid;
#Client should have thrown error
-$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
-like ( $psql_err, qr/error triggered for injection point memcontext-client-crash/);
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-crash/);
#Query the same process after detaching the injection point, using some other client and it should succeed.
-$node->safe_psql('postgres', "select injection_points_detach('memcontext-client-crash');");
-my $topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-crash');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
ok($topcontext_name = 'TopMemoryContext');
# Attaching to a target process injection point that throws an error
-$node->safe_psql('postgres', "select injection_points_attach('memcontext-server-crash', 'error');");
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-crash', 'error');");
#Server should have thrown error
-$node->psql('postgres', qq(select pg_get_process_memory_contexts($pid, true);), stderr => \$psql_err);
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
#Query the same process after detaching the injection point, using some other client and it should succeed.
-$node->safe_psql('postgres', "select injection_points_detach('memcontext-server-crash');");
-$topcontext_name = $node->safe_psql('postgres', "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-crash');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
ok($topcontext_name = 'TopMemoryContext');
done_testing();
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index aa898f025c0..5e3122a468b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1686,9 +1686,9 @@ MemoryContextCallback
MemoryContextCallbackFunction
MemoryContextCounters
MemoryContextData
+MemoryContextId
MemoryContextMethodID
MemoryContextMethods
-MemoryStatsContextId
MemoryStatsEntry
MemoryStatsDSHashEntry
MemoryStatsPrintFunc
--
2.39.3 (Apple Git-146)
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-12-12 09:46 Rahila Syed <[email protected]>
parent: torikoshia <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Rahila Syed @ 2025-12-12 09:46 UTC (permalink / raw)
To: torikoshia <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; pgsql-hackers
Hi,
Sorry for the late response. Thank you for your reviewing and testing the
patch.
On Mon, Dec 8, 2025 at 6:56 AM torikoshia <[email protected]>
wrote:
> On 2025-11-28 18:22, Rahila Syed wrote:
>
> Hi,
>
> > I'm attaching the updated patches, which primarily include cleanup and
> > have been rebased
> > following the CFbot report.
>
> Thanks for updating the patch!
>
> I observed an assertion failure when forcing a timeout as follows:
>
>
Good catch. This assertion is no longer valid because of recent updates
that reset the client_keys slot for a request when the client exits with a
timeout.
To address this, I’ve replaced the assertion with a check for -1 and now
return
from the function in that case.
It might be good to also document in func-admin.sgml that the function
> times out after 5 seconds when the target backend does not respond, and
> that in such a case NULLs are returned.
>
>
Added this.
From the comment, it sounded to me as if the client executing
> pg_get_process_memory_contexts() might not create the DSA in some cases.
> Is it correct to assume that such a situation can happen?
> In [1], as a response to concerns about using DSA inside a CFI handler,
> you wrote that “all the dynamic shared memory needed to store the
> statistics is created and deleted in the client function”.
> So I understood that it would never create the DSA inside the CFI
> handler.
> If that understanding is correct, perhaps the comment should be reworded
> to make that clear.
>
>
Yes, your understanding is correct. I reworded the comment accordingly.
> + context_id_lookup =
> hash_create("pg_get_remote_backend_memory_contexts",
>
> This appears to use the old function name. Should this be updated to
> "pg_get_process_memory_contexts" instead?
>
>
Modified this.
I will post the updated patch in response to Daniel's message that follows
your email.
Thank you,
Rahila Syed
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-12-12 11:04 Rahila Syed <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-12-12 11:04 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: pgsql-hackers; torikoshia <[email protected]>
Hi Daniel,
>
> Thanks for the patch, below are a few comments and suggestions. As I was
> reviewing I tweaked the below and have attached the comments as changes in
> 0003.
>
Thank you for the improvements.
All your changes look good to me. I have incorporated those in the v44
patch.
> + * Entry has been deleted due to client process exit. Make sure that
> the
> + * client always deletes the entry after taking required lock or this
> + * function may end up writing to unallocated memory.
> Can you explain this a bit further, I'm not sure I get it. The code goes
> on to
> release a lock immediately and then destroys the hash. Who is responsible
> for
> destroying the entry?
>
This just points to the general requirements of taking a lock before
writing to a shared variable.
This serves as a warning to other processes not to delete the entry without
taking a lock, since
we are about to write to the entry.
> == In ProcessGetMemoryContextInterrupt()
> I'm not a fan of having to exit's from the function doing duplicative
> cleanups,
> in the attached I've wrapped them in a conditional to just have one exit
> path.
> What do you think about that?
>
I agree with your approach. It certainly makes the code more concise and
easier to read.
> == In PublishMemoryContext()
> + /* Allocate DSA memory for storing path information */
> This comment is no longer accurate is it? The DSA has already been
> allocated
> at this point.
>
>
Yes, it is not valid anymore. Fixed accordingly.
Apart from this, I cleaned up the test module by removing unnecessary sql
functions, added some more injection points based tests and a few
minor tweaks.
Please find attached updated and rebased patches.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v44-0001-Add-function-to-report-memory-context-statistics.patch (57.7K, ../../CAH2L28vK5T_+u4hOvxm+z_9povNPQWMg=ZxtdD00hngY2QibWA@mail.gmail.com/3-v44-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From 4b20c9f6da3c9d92d3af85ddbb761843eb3d04b1 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 159 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1005 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 8 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
25 files changed, 1292 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 1b465bc8ba7..3e849a132e9 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,132 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5 seconds,
+ function returns NULL.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +428,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 2d946d6d9e9..7b40bac5f57 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -657,6 +657,17 @@ LANGUAGE INTERNAL
STRICT VOLATILE PARALLEL UNSAFE
AS 'pg_replication_origin_session_setup';
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -782,6 +793,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -808,6 +820,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 1bd3924e35e..baba657904b 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 7f8cf1fa2ec..749e68553b9 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index ba63b84dfc5..29454b8bf8a 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 3a65d841725..b89617d78db 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -871,6 +871,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index e7e4d652f97..eb86648f7b7 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index b23d0c19360..a5ed58a18c5 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -140,6 +141,7 @@ CalculateShmemSize(void)
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
InjectionPointShmemInit();
AioShmemInit();
WaitLSNShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index ebc3f4ca457..27b3b51cf2d 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 7dd75a490aa..e726f40dfbb 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3539,6 +3539,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index f39830dbb34..3889228b1ed 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -162,6 +162,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -405,6 +406,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 46dfb3dd133..c3ec83bd1ed 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -17,22 +17,130 @@
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Lock to control access to client_keys array */
+static LWLock *client_keys_lock = NULL;
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static void memstats_client_key_reset(int ProcNumber);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -143,24 +251,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +266,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -305,3 +428,845 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Check if the server process slot is not empty and exit early Non-empty
+ * slot means some other client backend is requesting the statistics from
+ * the same server process.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] != -1)
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(client_keys_lock);
+ return;
+ }
+ else
+ {
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+ }
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table. Here we just attach to those.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ INJECTION_POINT("memcontext-server-injection", NULL);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking
+ * required lock or this function may end up writing to unallocated
+ * memory.
+ */
+ if (!found || entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 4ed69ac7ba2..c5a36dcbc95 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -658,6 +658,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..31c4de9f0b4 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1008,6 +1008,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fd9448ec7b9..bef24d625d9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8617,6 +8617,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 9a7d733ddef..b76f24baed6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 5b0ce383408..613e769c84e 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -136,3 +136,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7bbe5a36959..617de0ebf91 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -19,7 +19,6 @@
#include "nodes/memnodes.h"
-
/*
* MaxAllocSize, MaxAllocHugeSize
* Quasi-arbitrary limits on size of allocations.
@@ -319,4 +318,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 0411db832f1..3799ef7c862 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..c9da4fc8c90 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9dd65b10254..eb25f426cf2 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1690,6 +1690,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] v44-0002-Test-module-to-test-memory-context-reporting-wit.patch (8.8K, ../../CAH2L28vK5T_+u4hOvxm+z_9povNPQWMg=ZxtdD00hngY2QibWA@mail.gmail.com/4-v44-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From b2315034628f0ae6572f9a4887a54f14974105c2 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 29 ++++
.../t/001_memcontext_inj.pl | 150 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 4 +
5 files changed, 196 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4c6d56d97d8..1156d731014 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..0a2dfc44f1c
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,29 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..b491d6ebc0a
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,150 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+#Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+#Server should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning for
+# one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+#Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+ok($psql_out = 'NULL');
+#Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+#Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test if the monitoring works fine, when the client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+#Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+#Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-12-18 19:54 Rahila Syed <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-12-18 19:54 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: pgsql-hackers; torikoshia <[email protected]>
Hi,
PFA the updated and rebased patches.
To summarize the work done in this thread,
here are some key concerns discussed in threads [1] and [2] and steps taken
to address those:
*DSA APIs and CFI Handler Safety*: DSA APIs, being high-level, are unsafe
to call from the CFI handler,
which can be invoked from low-level code. This concern was particularly
raised for APIs like `dsa_allocate()`
and `dsa_create()`.
To resolve this, these APIs have been moved out of the CFI handler
function. Now, the dynamic shared memory
needed to store the statistics is allocated and deleted in the client
function. The only operation performed in the CFI
handler is `dsm_attach()`, which attaches to DSA for copying statistics.
Since dsm_attach() only maps the existing
DSM into the current process address space and does not create a new DSM, I
don't see any specific reason why
it would be unsafe to call it from the CFI handler.
*Memory Leak in TopMemoryContext*: A memory leak was reported in the
TopMemoryContext and there were
concerns that memory allocated while executing the memory statistics
reporting function could impact its output.
To address this, we create an exclusive memory context under the NULL
context to handle all memory allocations in
`ProcessGetMemoryContextInterrupt`. This context does not fall under the
TopMemoryContext tree, ensuring that
allocations do not affect the function's outcome. The memory context is
reset at the end of the function, preventing
leaks.
*Error Reported in Thread [2]*: This issue has been fixed by switching to a
NULL resource owner before attaching
to DSM in the CFI handler.
*Other Improvements*:
1. Simplified the user interface by removing the `timeout` argument and
using a constant value instead.
2. Provided a default for the `get_summary` argument so users do not need
to pass a value if they choose not to.
3. The dsm_registry APIs are used to create and attach to DSA and DSHASH
tables, which helps avoid code duplication.
4. Replaced the static shared memory array with a DSHASH table, which holds
metadata such as pointers to memory
containing statistics for each process.
5. The updates previously made to mcxt.c have been moved to mcxtfuncs.c,
which now includes all the existing memory
statistics functions as well as the code for the new proposed function
6. One function that relies on an unexported API is added in mcxt.c
Thank you,
Rahila Syed
[1]. PostgreSQL: Re: pgsql: Add function to get memory context stats for
processes
<https://www.postgresql.org/message-id/CA%2BTgmoaey-kOP1k5FaUnQFd1fR0majVebWcL8ogfLbG_nt-Ytg%40mail.g...;
[2]. PostgreSQL: Re: Prevent an error on attaching/creating a DSM/DSA from
an interrupt handler.
<https://www.postgresql.org/message-id/flat/8B873D49-E0E5-4F9F-B8D6-CA4836B825CD%40yesql.se#7026d2fe4...;
Attachments:
[application/octet-stream] v45-0001-Add-function-to-report-memory-context-statistics.patch (57.9K, ../../CAH2L28tUUf8o0Ec-5d8XPe=hno3A9Ob4nQ7TgrM1Qt_vJ0Oo2Q@mail.gmail.com/3-v45-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From 024a1ce3cf4a75d025ecf6f906beac3b3d634bb2 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 159 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1009 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 8 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
25 files changed, 1296 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 2896cd9e429..5eac0e5f73c 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,132 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5 seconds,
+ function returns NULL.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +428,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 2d946d6d9e9..7b40bac5f57 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -657,6 +657,17 @@ LANGUAGE INTERNAL
STRICT VOLATILE PARALLEL UNSAFE
AS 'pg_replication_origin_session_setup';
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -782,6 +793,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -808,6 +820,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 1bd3924e35e..baba657904b 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 7f8cf1fa2ec..749e68553b9 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -679,6 +679,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index ba63b84dfc5..29454b8bf8a 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 3a65d841725..b89617d78db 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -871,6 +871,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index e7e4d652f97..eb86648f7b7 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index b23d0c19360..a5ed58a18c5 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -140,6 +141,7 @@ CalculateShmemSize(void)
size = add_size(size, SlotSyncShmemSize());
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
InjectionPointShmemInit();
AioShmemInit();
WaitLSNShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 087821311cc..8963285cc12 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -691,6 +691,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index ebc3f4ca457..27b3b51cf2d 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 7dd75a490aa..e726f40dfbb 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3539,6 +3539,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index c0632bf901a..bf75b891495 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -162,6 +162,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -405,6 +406,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 46dfb3dd133..babf8706513 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -17,22 +17,130 @@
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Lock to control access to client_keys array */
+static LWLock *client_keys_lock = NULL;
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static void memstats_client_key_reset(int ProcNumber);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -143,24 +251,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +266,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -305,3 +428,849 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Check if the server process slot is not empty and exit early Non-empty
+ * slot means some other client backend is requesting the statistics from
+ * the same server process.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] != -1)
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+ ResourceOwner currentOwner;
+
+ PublishMemoryContextPending = false;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(client_keys_lock);
+ return;
+ }
+ else
+ {
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+ }
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ currentOwner = CurrentResourceOwner;
+ CurrentResourceOwner = NULL;
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table. Here we just attach to those.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+ CurrentResourceOwner = currentOwner;
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ INJECTION_POINT("memcontext-server-injection", NULL);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking
+ * required lock or this function may end up writing to unallocated
+ * memory.
+ */
+ if (!found || entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 4ed69ac7ba2..c5a36dcbc95 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -658,6 +658,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 47fd774c7d2..31c4de9f0b4 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1008,6 +1008,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fd9448ec7b9..bef24d625d9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8617,6 +8617,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 9a7d733ddef..b76f24baed6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 5b0ce383408..613e769c84e 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -136,3 +136,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index afeeb1ca019..345d5a0ecb1 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7bbe5a36959..617de0ebf91 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -19,7 +19,6 @@
#include "nodes/memnodes.h"
-
/*
* MaxAllocSize, MaxAllocHugeSize
* Quasi-arbitrary limits on size of allocations.
@@ -319,4 +318,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 0411db832f1..3799ef7c862 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..c9da4fc8c90 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 04845d5e680..6b964360718 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1700,6 +1700,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] v45-0002-Test-module-to-test-memory-context-reporting-wit.patch (8.8K, ../../CAH2L28tUUf8o0Ec-5d8XPe=hno3A9Ob4nQ7TgrM1Qt_vJ0Oo2Q@mail.gmail.com/4-v45-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From e9c37a3f67f00821b998aec8443797c710703474 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 29 ++++
.../t/001_memcontext_inj.pl | 150 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 4 +
5 files changed, 196 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4c6d56d97d8..1156d731014 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..0a2dfc44f1c
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,29 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..b491d6ebc0a
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,150 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+#Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+#Server should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning for
+# one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+#Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+ok($psql_out = 'NULL');
+#Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+#Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test if the monitoring works fine, when the client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+#Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+#Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2025-12-29 08:56 Rahila Syed <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2025-12-29 08:56 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
Hi,
I've included some additional description inline and attached rebased
patches after
CFbot reported a conflict.
> *DSA APIs and CFI Handler Safety*: DSA APIs, being high-level, are unsafe
> to call from the CFI handler,
> which can be invoked from low-level code. This concern was particularly
> raised for APIs like `dsa_allocate()`
> and `dsa_create()`.
> To resolve this, these APIs have been moved out of the CFI handler
> function. Now, the dynamic shared memory
> needed to store the statistics is allocated and deleted in the client
> function. The only operation performed in the CFI
> handler is `dsm_attach()`, which attaches to DSA for copying statistics.
> Since dsm_attach() only maps the existing
> DSM into the current process address space and does not create a new DSM,
> I don't see any specific reason why
> it would be unsafe to call it from the CFI handler.
>
>
Following are the details about the use of DSM in the patch:
- DSA Creation: A Dynamic Shared Area (DSA) is used to store memory context
statistics.
- Client Process: When fetching memory context statistics, the client
allocates a 1 MB chunk
in the DSA, reads the statistics from the memory chunk, copies it into a
tuple store, and then
frees the chunk.
- Storage: Pointers to these chunks are stored in a DSHASH table indexed by
the client’s
proc number. Each entry in the DSHASH table also stores additional
metadata related to the
client’s request.
- Attachment: Backends only attach to the DSM segments for the DSA and
DSHASH table
when necessary i.e when a process queries memory context statistics or is
queried by
another backend.
Once attached, they remain so until the session ends, at which point they
remove their DSHASH
entry if any and detach from DSA and DSHASH segments.
- Lifecycle: The DSA and DSHASH structures are created upon the first SQL
function invocation
and destroyed on server restart.
> *Error Reported in Thread [2]*: This issue has been fixed by switching to
> a NULL resource owner before attaching
> to DSM in the CFI handler.
>
>
This error mentioned in thread [2] is triggered during CFI() call from
secure_read() when a
backend is waiting for commands and it has an open transaction which is
going to abort
Below are some details about this fix.
It is safe to temporarily set the resource owner to NULL before attaching
to the DSA
and DSHASH, since these segments are intended to be attached for the full
session
and are detached only when the session ends.
We also restore the original resource owner immediately after the attach
completes.
Other possible fixes include:
1.Adjusting resource‑owner behavior
Either allow resource‑owner enlargement during release, or delay marking it
as releasing until
the abort actually begins.
2. Updating DSM registry APIs (e.g., GetNamedDSA)
Detect when the current resource owner is in a releasing state and
temporarily set
CurrentResourceOwner to NULL before calling dsa_attach.
3. Handling it in the DSA layer
This was discussed in thread [2], but concerns were raised that DSA should
not compensate
for incorrect caller state; the caller must ensure the resource owner is
valid.
Kindly let me know your views.
Thank you,
Rahila Syed
[2]. PostgreSQL: Re: Prevent an error on attaching/creating a DSM/DSA from
an interrupt handler
<https://www.postgresql.org/message-id/flat/8B873D49-E0E5-4F9F-B8D6-CA4836B825CD%40yesql.se#7026d2fe4...;
Attachments:
[application/octet-stream] v46-0002-Test-module-to-test-memory-context-reporting-wit.patch (8.8K, ../../CAH2L28vF7R08NUo6Sme8LydRaFcKhS6fu0psfxUQoxsVVqJMwA@mail.gmail.com/3-v46-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From 6e1e38079e24d1d2fb96f013ee1ca04424564a18 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 29 ++++
.../t/001_memcontext_inj.pl | 150 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 4 +
5 files changed, 196 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4c6d56d97d8..1156d731014 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..0a2dfc44f1c
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,29 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..b491d6ebc0a
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,150 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+#Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+#Server should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning for
+# one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+#Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+ok($psql_out = 'NULL');
+#Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+#Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test if the monitoring works fine, when the client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+#Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+#Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
[application/octet-stream] v46-0001-Add-function-to-report-memory-context-statistics.patch (57.9K, ../../CAH2L28vF7R08NUo6Sme8LydRaFcKhS6fu0psfxUQoxsVVqJMwA@mail.gmail.com/4-v46-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From 6b97338f59f073cadbff4d086f2a062b770f9280 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 159 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1009 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 8 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
25 files changed, 1296 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 2896cd9e429..5eac0e5f73c 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,132 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5 seconds,
+ function returns NULL.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +428,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 2d946d6d9e9..7b40bac5f57 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -657,6 +657,17 @@ LANGUAGE INTERNAL
STRICT VOLATILE PARALLEL UNSAFE
AS 'pg_replication_origin_session_setup';
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -782,6 +793,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -808,6 +820,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 1bd3924e35e..baba657904b 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 2eac8ac30d3..a22f77b3673 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -685,6 +685,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index ba63b84dfc5..29454b8bf8a 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 3a65d841725..b89617d78db 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -871,6 +871,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index 27e86cf393f..7149a67fcbc 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index e7e4d652f97..eb86648f7b7 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index adebba625e6..a68721b6f4d 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -141,6 +142,7 @@ CalculateShmemSize(void)
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
size = add_size(size, LogicalDecodingCtlShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -330,6 +332,7 @@ CreateOrAttachShmemStructs(void)
AioShmemInit();
WaitLSNShmemInit();
LogicalDecodingCtlShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index b0b93d96091..ac4d08ce422 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -694,6 +694,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index ebc3f4ca457..27b3b51cf2d 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 7dd75a490aa..e726f40dfbb 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3539,6 +3539,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index dcfadbd5aae..4df4f8ae121 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -162,6 +162,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -406,6 +407,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 46dfb3dd133..babf8706513 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -17,22 +17,130 @@
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Lock to control access to client_keys array */
+static LWLock *client_keys_lock = NULL;
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static void memstats_client_key_reset(int ProcNumber);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -143,24 +251,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +266,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -305,3 +428,849 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Check if the server process slot is not empty and exit early Non-empty
+ * slot means some other client backend is requesting the statistics from
+ * the same server process.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] != -1)
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+ ResourceOwner currentOwner;
+
+ PublishMemoryContextPending = false;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(client_keys_lock);
+ return;
+ }
+ else
+ {
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+ }
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ currentOwner = CurrentResourceOwner;
+ CurrentResourceOwner = NULL;
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table. Here we just attach to those.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+ CurrentResourceOwner = currentOwner;
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ INJECTION_POINT("memcontext-server-injection", NULL);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking
+ * required lock or this function may end up writing to unallocated
+ * memory.
+ */
+ if (!found || entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index d31cb45a058..92b0446b80c 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b7e94ca45bd..942ee5c34f4 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -661,6 +661,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 5c1a06d86fd..d601069b99f 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1011,6 +1011,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index fd9448ec7b9..bef24d625d9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8617,6 +8617,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index 9a7d733ddef..b76f24baed6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 533344509e9..8f83c8801a7 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -137,3 +137,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 8e428f298c6..1e5f5b1f957 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 7bbe5a36959..617de0ebf91 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -19,7 +19,6 @@
#include "nodes/memnodes.h"
-
/*
* MaxAllocSize, MaxAllocHugeSize
* Quasi-arbitrary limits on size of allocations.
@@ -319,4 +318,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 0411db832f1..3799ef7c862 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..c9da4fc8c90 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ceb3fc5d980..36d76262030 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1701,6 +1701,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-01-09 11:22 Rahila Syed <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2026-01-09 11:22 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
Hi,
>> *Error Reported in Thread [2]*: This issue has been fixed by switching
>> to a NULL resource owner before attaching
>> to DSM in the CFI handler.
>>
>>
> This error mentioned in thread [2] is triggered during CFI() call from
> secure_read() when a
> backend is waiting for commands and it has an open transaction which is
> going to abort
>
> Below are some details about this fix.
>
> It is safe to temporarily set the resource owner to NULL before attaching
> to the DSA
> and DSHASH, since these segments are intended to be attached for the full
> session
> and are detached only when the session ends.
> We also restore the original resource owner immediately after the attach
> completes.
>
>
After further discussion and reviewing Robert's email[1] on this topic, a
safer solution
is to avoid running ProcessGetMemoryContextInterrupt during an aborted
transaction.
This should help prevent additional errors when the transaction is already
in error handling
state. Also, reporting memory context statistics from an aborting
transaction won't
be very useful as some of that memory usage won't be valid after abort
completes.
Attached is the updated patch that addresses this.
> Other possible fixes include:
> 1.Adjusting resource‑owner behavior
> Either allow resource‑owner enlargement during release, or delay marking
> it as releasing until
> the abort actually begins.
>
Sorry, this point is invalid as resource-owner is already being marked as
releasing from
AbortTransaction.
Thank you,
Rahila Syed
[1].
https://www.postgresql.org/message-id/CA%2BTgmoapJ6erjT21uPO12wTtoOmj6w-dp6T3qySN%2BNSc1cdEKw%40mail...
>
Attachments:
[application/octet-stream] v47-0002-Test-module-to-test-memory-context-reporting-wit.patch (8.8K, ../../CAH2L28vqKDTogugPnr54hY8k6enP02U_qMBUtnmo7CbA6OfhHQ@mail.gmail.com/3-v47-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From 83cf295b1c90fb43c7939c98f66e9aa4159d923f Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 29 ++++
.../t/001_memcontext_inj.pl | 150 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 4 +
5 files changed, 196 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4c6d56d97d8..1156d731014 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..0a2dfc44f1c
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,29 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..b491d6ebc0a
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,150 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+#Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+#Server should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning for
+# one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+#Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+ok($psql_out = 'NULL');
+#Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+#Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test if the monitoring works fine, when the client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+#Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+#Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
[application/octet-stream] v47-0001-Add-function-to-report-memory-context-statistics.patch (57.9K, ../../CAH2L28vqKDTogugPnr54hY8k6enP02U_qMBUtnmo7CbA6OfhHQ@mail.gmail.com/4-v47-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From 781a7649bf8fb5e89fb900664f9a8d05844eb8a9 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 159 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1012 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 8 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
25 files changed, 1299 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 2896cd9e429..5eac0e5f73c 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,132 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5 seconds,
+ function returns NULL.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +428,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index eb9e31ae1bf..d1416e5534d 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -657,6 +657,17 @@ LANGUAGE INTERNAL
STRICT VOLATILE PARALLEL UNSAFE
AS 'pg_replication_origin_session_setup';
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -782,6 +793,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -808,6 +820,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 3e507d23cc9..fbebe506495 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -791,6 +791,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 6482c21b8f9..1b7d5e7ffdc 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -685,6 +685,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index a2c0ff012c5..cb18ce80893 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 1a20387c4bd..3ac0fba225a 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -871,6 +871,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index a1a4f65f9a9..18a9e7f85e1 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c3d56c866d3..49dc3e88023 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 85c67b2c183..211199e985a 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -141,6 +142,7 @@ CalculateShmemSize(void)
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
size = add_size(size, LogicalDecodingCtlShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -330,6 +332,7 @@ CreateOrAttachShmemStructs(void)
AioShmemInit();
WaitLSNShmemInit();
LogicalDecodingCtlShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 8e56922dcea..601e01d2574 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -694,6 +694,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 66274029c74..61957fc9b74 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e54bf1e760f..199245e4c1f 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3539,6 +3539,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 3299de23bb3..6f78075bdfc 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -163,6 +163,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -407,6 +408,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 12b8d4cefaf..335fdcdc329 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -17,22 +17,130 @@
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Lock to control access to client_keys array */
+static LWLock *client_keys_lock = NULL;
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static void memstats_client_key_reset(int ProcNumber);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -143,24 +251,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +266,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -305,3 +428,852 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Check if the server process slot is not empty and exit early Non-empty
+ * slot means some other client backend is requesting the statistics from
+ * the same server process.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] != -1)
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Avoid performing any shared memory operations in aborted transaction,
+ * the caller will get the fallback behaviour of the past known stats.
+ */
+ if (IsAbortedTransactionBlockState())
+ return;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(client_keys_lock);
+ return;
+ }
+ else
+ {
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+ }
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table. Here we just attach to those.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ INJECTION_POINT("memcontext-server-injection", NULL);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking
+ * required lock or this function may end up writing to unallocated
+ * memory.
+ */
+ if (!found || entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..e246fc7ce1f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..7e6de81e7d6 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -661,6 +661,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 073bdb35d2a..fd75345ff5f 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1011,6 +1011,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 2ac69bf2df5..ef50383ff4f 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8633,6 +8633,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..7e301affccc 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index 94f818b9f10..ed249e72a26 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -137,3 +137,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index e52b8eb7697..4f1bcbf709f 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 2bc13c3a054..435a2a27c94 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -19,7 +19,6 @@
#include "nodes/memnodes.h"
-
/*
* MaxAllocSize, MaxAllocHugeSize
* Quasi-arbitrary limits on size of allocations.
@@ -319,4 +318,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 0411db832f1..3799ef7c862 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 66179f026b3..c9da4fc8c90 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 09e7f1d420e..c32d56c0c99 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1701,6 +1701,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-01-13 19:31 Robert Haas <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 2 replies; 65+ messages in thread
From: Robert Haas @ 2026-01-13 19:31 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
On Fri, Jan 9, 2026 at 6:22 AM Rahila Syed <[email protected]> wrote:
> After further discussion and reviewing Robert's email[1] on this topic, a safer solution
> is to avoid running ProcessGetMemoryContextInterrupt during an aborted transaction.
> This should help prevent additional errors when the transaction is already in error handling
> state. Also, reporting memory context statistics from an aborting transaction won't
> be very useful as some of that memory usage won't be valid after abort completes.
> Attached is the updated patch that addresses this.
I think the question here is whether it's safe to interrupt the server
at an arbitrary CFI to run ProcessGetMemoryContextInterrupt. The good
news is that we now know that that function won't be executed (or at
least not in a substantive way) when we're an aborted transaction. But
we have no guarantee that we're in a transaction at all, which I think
is concerning, because it seems hard to write code that behaves
properly both inside of a transaction and outside of a transaction.
Some things I notice reading through:
+ LWLockAcquire(client_keys_lock, LW_SHARED);
LWLocks normally use NamesLikeThis not names_like_this and are
generally created by just listing them in lwlocklist.h.
Also, it's not really safe to acquire an LWLock if there's no
transaction active. If we error afterward, what will release the
LWLock?
+ else
+ {
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+ }
The "else" is not really necessary here because the "if" portion ends
with "return".
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+
"publish_memory_context_statistics",
+
ALLOCSET_SMALL_SIZES);
The comments do a good job justifying this, but as far as I know it
would be the only instance of this pattern in the entire source tree.
Are we really sure we want to deviate from the idea of having the
memory context tree be a tree? And is it really so bad if the memory
used to report memory contexts is included in the output?
+ MemoryStatsDsaArea =
GetNamedDSA("memory_context_statistics_dsa",
+
&found);
...
+ MemoryStatsDsHash =
GetNamedDSHash("memory_context_statistics_dshash",
+
&memctx_dsh_params, &found);
...
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
Like LWLockAcquire, all of this code supposes that there's a
transaction available to manage resource acquisition and release and
to clean up after errors. I doubt that any of this is safe without a
transaction.
I think my view overall here is: bailing out when the current
transaction is aborted is a good start in terms of making this safe,
but I don't think it's enough. If we get here when the backend is
idle, for example, which seems like it would be possible, then we'll
have no active transaction, and then I think that a lot of what's
being called here is not going to necessarily work as intended. There
are other places in the source code that do various push-ups to get
around this problem -- e.g. look at RemoveTempRelationsCallback, which
knows that it will be called from a transaction but not whether that
transaction will be aborted. I don't think that's the approach you
want here: restarting an aborted transaction in this context wouldn't
be smart. But spinning one up if there isn't one might be reasonable.
See ProcessCatchupInterrupt() for an example of existing code that
deals with this problem -- again that's probably not quite the same
situation, but it gives you an idea what other people have done
before.
One other point to note here is that if, by some chance, a failure
occurs as a result of receiving a memory-context interrupt, it's going
to fail the currently-running transaction. Unlike the problem
discussed in the previous paragraph, that's not a system-integrity
issue: from the point of view of PostgreSQL's transaction system,
having process #1 cause process #2's transaction to fail is 100% OK
and nothing will break. From the user's point of view, however, this
will be very painful: you just wanted to inspect the state of a
running transaction, very possibly a long-running one that had done a
lot of work, and you accidentally killed it. So I think it's going to
be important that such cases are extremely rare in practice. If they
happen because of extremely weird scenarios like somebody manually
removing a shared memory segment while it's still in use by
PostgreSQL, or the system running out of memory at exactly the wrong
moment, I think that's probably fine. If they happen because of
something like a low-probability race condition, that's going to be
unacceptable to users. Nobody is going to be happy about using the
feature if it can cause the transactions that they're querying to
randomly die, even if the chances are low. I'm not saying the patch
actually has this problem, just that it's something to think very
carefully about.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-01-13 20:26 Andres Freund <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: Andres Freund @ 2026-01-13 20:26 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Rahila Syed <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
Hi,
On 2026-01-13 14:31:00 -0500, Robert Haas wrote:
> Also, it's not really safe to acquire an LWLock if there's no
> transaction active. If we error afterward, what will release the
> LWLock?
All the error handling paths (hopefully) have an LWLockReleaseAll()... Which
is pretty crucial given that we do stuff outside of transactions in other
places.
That doesn't mean the other concerns about resource management are unfounded,
however.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-01-13 21:11 Robert Haas <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Robert Haas @ 2026-01-13 21:11 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Rahila Syed <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
On Tue, Jan 13, 2026 at 3:26 PM Andres Freund <[email protected]> wrote:
> All the error handling paths (hopefully) have an LWLockReleaseAll()... Which
> is pretty crucial given that we do stuff outside of transactions in other
> places.
>
> That doesn't mean the other concerns about resource management are unfounded,
> however.
Yeah, I actually wasn't completely sure about that particular comment.
I think what will happen if we ERROR outside of a transaction is that
it will become FATAL and kill the backend, but I'm not 100% positive
about that.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-01-13 21:47 Andres Freund <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Andres Freund @ 2026-01-13 21:47 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Rahila Syed <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
Hi,
On 2026-01-13 16:11:24 -0500, Robert Haas wrote:
> On Tue, Jan 13, 2026 at 3:26 PM Andres Freund <[email protected]> wrote:
> > All the error handling paths (hopefully) have an LWLockReleaseAll()... Which
> > is pretty crucial given that we do stuff outside of transactions in other
> > places.
> >
> > That doesn't mean the other concerns about resource management are unfounded,
> > however.
>
> Yeah, I actually wasn't completely sure about that particular comment.
> I think what will happen if we ERROR outside of a transaction is that
> it will become FATAL and kill the backend, but I'm not 100% positive
> about that.
I'm pretty sure that doesn't generally happen. There's promotion to FATAL if
the top-level sigsetjmp() hasn't yet run (c.f. the check for
PG_exception_stack in errstart()), but once it has been reached, it stays
configured.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-01-14 13:26 Robert Haas <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Robert Haas @ 2026-01-14 13:26 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Rahila Syed <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
On Tue, Jan 13, 2026 at 4:47 PM Andres Freund <[email protected]> wrote:
> I'm pretty sure that doesn't generally happen. There's promotion to FATAL if
> the top-level sigsetjmp() hasn't yet run (c.f. the check for
> PG_exception_stack in errstart()), but once it has been reached, it stays
> configured.
All right, then I guess I don't fully understand how the
error-outside-of-a-transaction case is handled. But I still think that
code like this needs to run in a transaction to avoid unexpected and
undesirable results. Do you see it differently?
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-01-14 13:32 Andres Freund <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Andres Freund @ 2026-01-14 13:32 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Rahila Syed <[email protected]>; Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
Hi,
On 2026-01-14 08:26:24 -0500, Robert Haas wrote:
> On Tue, Jan 13, 2026 at 4:47 PM Andres Freund <[email protected]> wrote:
> > I'm pretty sure that doesn't generally happen. There's promotion to FATAL if
> > the top-level sigsetjmp() hasn't yet run (c.f. the check for
> > PG_exception_stack in errstart()), but once it has been reached, it stays
> > configured.
>
> All right, then I guess I don't fully understand how the
> error-outside-of-a-transaction case is handled.
For those we just rely on the error cleanup inside the top-level sigsetmp()
blocks... Which of course is a pretty random subset of cleanups that also
differs between process types, which is quite ... not great.
> But I still think that
> code like this needs to run in a transaction to avoid unexpected and
> undesirable results. Do you see it differently?
I'm waffling on it a bit, tbh. I think doing it for something like backends it
could make sense, but e.g. for something like checkpointer, that normally
never runs a transaction, it seems like a bad idea. Mainly because processes
that don't run transactions won't have an AbortCurrentTransaction() or such in
their top-level sigsetjmp() and thus would never abort the transaction that we
started, if there were an error while doing the reporting.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-01-19 17:31 Rahila Syed <[email protected]>
parent: Robert Haas <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2026-01-19 17:31 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
Hi,
Thank you for your feedback. Please find a few responses inline.
I am working on incorporating all the feedback in the patch and will share
it in a follow-up email.
> Some things I notice reading through:
>
> + LWLockAcquire(client_keys_lock, LW_SHARED);
>
> LWLocks normally use NamesLikeThis not names_like_this and are
> generally created by just listing them in lwlocklist.h.
>
>
I will fix it, accordingly.
> + else
> + {
> + clientProcNumber = client_keys[MyProcNumber];
> + client_keys[MyProcNumber] = -1;
> + LWLockRelease(client_keys_lock);
> + }
>
> The "else" is not really necessary here because the "if" portion ends
> with "return".
>
Fixed this in the attached patch.
>
> + memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
> +
> "publish_memory_context_statistics",
> +
> ALLOCSET_SMALL_SIZES);
>
> The comments do a good job justifying this, but as far as I know it
> would be the only instance of this pattern in the entire source tree.
> Are we really sure we want to deviate from the idea of having the
> memory context tree be a tree? And is it really so bad if the memory
> used to report memory contexts is included in the output?
>
>
This was implemented in response to a review suggestion [1].
If required, it can be updated to create one under CurrentMemoryContext.
+ MemoryStatsDsaArea =
> GetNamedDSA("memory_context_statistics_dsa",
> +
> &found);
> ...
> + MemoryStatsDsHash =
> GetNamedDSHash("memory_context_statistics_dshash",
> +
> &memctx_dsh_params, &found);
> ...
> + entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
>
> Like LWLockAcquire, all of this code supposes that there's a
> transaction available to manage resource acquisition and release and
> to clean up after errors. I doubt that any of this is safe without a
> transaction.
>
Starting a transaction from CFI works for a client backend but
not for auxiliary processes. When I try to execute StartTransaction()
from a checkpointer process, the following assertion fails,
Assert(MyProc->vxid.procNumber == vxid.procNumber);
This is because MyProc->vxid.procNumber is not set for auxiliary
processes.
Please find attached a rebased patch, that fixes a build error reported
on the commit-fest app.
[1]. PostgreSQL: Re: Prevent an error on attaching/creating a DSM/DSA from
an interrupt handler.
<https://www.postgresql.org/message-id/594293.1747708165%40sss.pgh.pa.us;
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v48-0002-Test-module-to-test-memory-context-reporting-wit.patch (8.8K, ../../CAH2L28u=78SynqRkNn_v0Xo=c7eiaQQ2b5QQ-pSOmMH-z52eqQ@mail.gmail.com/3-v48-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From 56ee69ad6c2a9a1e6cd3c22d8c2bbb3d46a39127 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 29 ++++
.../t/001_memcontext_inj.pl | 150 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 4 +
5 files changed, 196 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4c6d56d97d8..1156d731014 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..0a2dfc44f1c
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,29 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..b491d6ebc0a
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,150 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+#Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+#Server should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning for
+# one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+#Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+ok($psql_out = 'NULL');
+#Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+#Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test if the monitoring works fine, when the client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+#Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+#Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
[application/octet-stream] v48-0001-Add-function-to-report-memory-context-statistics.patch (58.0K, ../../CAH2L28u=78SynqRkNn_v0Xo=c7eiaQQ2b5QQ-pSOmMH-z52eqQ@mail.gmail.com/4-v48-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From a2743b93abc6c05c84b544155027a878734351c7 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 159 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1013 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 8 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
25 files changed, 1300 insertions(+), 21 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index e7ea16f73b3..e6b9d004ac4 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,132 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5 seconds,
+ function returns NULL.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +428,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index eb9e31ae1bf..d1416e5534d 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -657,6 +657,17 @@ LANGUAGE INTERNAL
STRICT VOLATILE PARALLEL UNSAFE
AS 'pg_replication_origin_session_setup';
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -782,6 +793,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -808,6 +820,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 22379de1e31..a00bac51264 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -788,6 +788,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 6482c21b8f9..1b7d5e7ffdc 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -685,6 +685,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index a2c0ff012c5..cb18ce80893 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 1a20387c4bd..3ac0fba225a 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -871,6 +871,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index a1a4f65f9a9..18a9e7f85e1 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c3d56c866d3..49dc3e88023 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 85c67b2c183..211199e985a 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -141,6 +142,7 @@ CalculateShmemSize(void)
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
size = add_size(size, LogicalDecodingCtlShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -330,6 +332,7 @@ CreateOrAttachShmemStructs(void)
AioShmemInit();
WaitLSNShmemInit();
LogicalDecodingCtlShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 8e56922dcea..601e01d2574 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -694,6 +694,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 063826ae576..0fdfa59d0b2 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index e54bf1e760f..199245e4c1f 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3539,6 +3539,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 5537a2d2530..f6e8feac6c8 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -163,6 +163,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -410,6 +411,7 @@ SubtransSLRU "Waiting to access the sub-transaction SLRU cache."
XactSLRU "Waiting to access the transaction status SLRU cache."
ParallelVacuumDSA "Waiting for parallel vacuum dynamic shared memory allocation."
AioUringCompletion "Waiting for another process to complete IO via io_uring."
+MemoryContextReportingKeys "Waiting for another process to complete reading or writing the memory reporting keys."
# No "ABI_compatibility" region here as WaitEventLWLock has its own C code.
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 12b8d4cefaf..472ee1b96e3 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,24 +15,133 @@
#include "postgres.h"
+#include "access/xact.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Lock to control access to client_keys array */
+static LWLock *client_keys_lock = NULL;
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static void memstats_client_key_reset(int ProcNumber);
+static const char *ContextTypeToString(NodeTag type);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
+static List *compute_context_path(MemoryContext c, HTAB *context_id_lookup);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -143,24 +252,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = ContextTypeToString(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -175,6 +267,38 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
list_free(path);
}
+/*
+ * ContextTypeToString
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+const char *
+ContextTypeToString(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
/*
* pg_get_backend_memory_contexts
* SQL SRF showing backend memory context.
@@ -305,3 +429,852 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Check if the server process slot is not empty and exit early Non-empty
+ * slot means some other client backend is requesting the statistics from
+ * the same server process.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] != -1)
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(client_keys_lock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(client_keys_lock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ int path_length;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(ContextTypeToString(memcxt_info[i].type));
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+ if (memcxt_info[i].path[0] != 0)
+ {
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(client_keys_lock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize() + sizeof(LWLockPadded), &found);
+ client_keys_lock = (LWLock *) ((char *) client_keys + MemoryContextKeysShmemSize());
+
+ if (!found)
+ {
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+ LWLockInitialize(client_keys_lock, LWTRANCHE_MEMORY_CONTEXT_KEYS);
+ }
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size sz = 0;
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+ sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
+
+ return sz;
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Avoid performing any shared memory operations in aborted transaction,
+ * the caller will get the fallback behaviour of the past known stats.
+ */
+ if (IsAbortedTransactionBlockState())
+ return;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(client_keys_lock, LW_SHARED);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(client_keys_lock);
+ return;
+ }
+ else
+ {
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(client_keys_lock);
+ }
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table. Here we just attach to those.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ INJECTION_POINT("memcontext-server-injection", NULL);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking
+ * required lock or this function may end up writing to unallocated
+ * memory.
+ */
+ if (!found || entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ elog(ERROR, "hash table corrupted, can't construct path value");
+
+ path = lcons_int(cur_entry->context_id, path);
+ }
+
+ return path;
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(client_keys_lock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(client_keys_lock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..e246fc7ce1f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..7e6de81e7d6 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -661,6 +661,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 073bdb35d2a..fd75345ff5f 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1011,6 +1011,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 894b6a1b6d6..c213211a6b5 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8633,6 +8633,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..7e301affccc 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index e94ebce95b9..74b611b9e08 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -137,3 +137,4 @@ PG_LWLOCKTRANCHE(SUBTRANS_SLRU, SubtransSLRU)
PG_LWLOCKTRANCHE(XACT_SLRU, XactSLRU)
PG_LWLOCKTRANCHE(PARALLEL_VACUUM_DSA, ParallelVacuumDSA)
PG_LWLOCKTRANCHE(AIO_URING_COMPLETION, AioUringCompletion)
+PG_LWLOCKTRANCHE(MEMORY_CONTEXT_KEYS, MemoryContextReportingKeys)
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index e52b8eb7697..4f1bcbf709f 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 2bc13c3a054..435a2a27c94 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -19,7 +19,6 @@
#include "nodes/memnodes.h"
-
/*
* MaxAllocSize, MaxAllocHugeSize
* Quasi-arbitrary limits on size of allocations.
@@ -319,4 +318,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3dd63fd88ed..c84e5adce16 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 004f9a70e00..d34d2b6211c 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 3f3a888fd0e..d057727faa0 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1706,6 +1706,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-01-19 18:02 Robert Haas <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Robert Haas @ 2026-01-19 18:02 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
On Mon, Jan 19, 2026 at 12:31 PM Rahila Syed <[email protected]> wrote:
> Starting a transaction from CFI works for a client backend but
> not for auxiliary processes. When I try to execute StartTransaction()
> from a checkpointer process, the following assertion fails,
> Assert(MyProc->vxid.procNumber == vxid.procNumber);
> This is because MyProc->vxid.procNumber is not set for auxiliary
> processes.
Unfortunately, somebody is going to need to think through - and
perhaps test - what happens in each individual type of background
process -- not just auxiliary processes but also background workers,
including but not limited to parallel workers. Some of them have error
recovery logic that is similar to transaction cleanup but only covers
a subset of things, in which case the question will be whether that
logic handles all the kinds of resources that this code might acquire.
Some of them may just straight up kill the process if an error occurs,
which is fine for this patch as long as it only happens in extreme
situations (e.g. OOM).
In other words, we don't necessarily need a transaction specifically,
but we need whatever form of error recovery is in use in any given
process to be appropriate to the code that this patch proposes to
execute. For a regular backend, that's a transaction.
Also worth noting: I don't think that all backend types actually use
CHECK_FOR_INTERRUPTS(). For those that don't, other updates may be
needed to make this feature work.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-02-02 12:28 Rahila Syed <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2026-02-02 12:28 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers
Hi Robert, Daniel
Please find attached the updated patches which incorporate your feedback.
Following are responses to Robert's comments:
Unfortunately, somebody is going to need to think through - and
> perhaps test - what happens in each individual type of background
> process -- not just auxiliary processes but also background workers,
> including but not limited to parallel workers. Some of them have error
> recovery logic that is similar to transaction cleanup but only covers
> a subset of things, in which case the question will be whether that
> logic handles all the kinds of resources that this code might acquire.
> Some of them may just straight up kill the process if an error occurs,
> which is fine for this patch as long as it only happens in extreme
> situations (e.g. OOM).
>
I tested the resource cleanup in case of error, with respect to each of the
processes types i.e Auxiliary process/background workers/client backends
and below are my findings.
The only resources acquired within the ProcessGetMemoryContextInterrupt
function are of the kind dsm_resowner_desc through GetNamedDSA and
GetNamedDsHash.
However, these functions make the resource owner forget these resources
when they call dsm_pin_mapping. This is done to prevent them from
getting detached at the time of error handling or transaction abort/commit.
Because when a dsm resource is released, the associated dsm segment is
detached.
Since we want the DSM segments to remain mapped until a process
exits, they are not added to resource owners. This is done intentionally to
prevent repeatedly attaching or detaching the DSM segments, if the same
process is queried for statistics multiple times.
As no resources are tracked during the execution
ProcessGetMemoryContextInterrupt
irrespective of its execution within or outside a transaction, the error
handling
in both the cases does not process resource cleanup unless the process
exits.
During proc_exit, the cleanup/detach runs via dsm_backend_shutdown().
> + LWLockAcquire(client_keys_lock, LW_SHARED);
>
> LWLocks normally use NamesLikeThis not names_like_this and are
> generally created by just listing them in lwlocklist.h.
>
Fixed.
Also, it's not really safe to acquire an LWLock if there's no
> transaction active. If we error afterward, what will release the
> LWLock?
I verified that when an error occurs outside of a transaction in various
types of processes, LWLockReleaseAll() is called during error handling.
1. For auxiliary processes - called within sigsetjmp() from main functions.
2. For client backend - called during process exit as error is converted to
"FATAL: terminating connection because protocol synchronization was lost"
3. For a background worker like parallel worker - called during process
exit,
as the worker is killed on encountering an error.
+ else
> + {
> + clientProcNumber = client_keys[MyProcNumber];
> + client_keys[MyProcNumber] = -1;
> + LWLockRelease(client_keys_lock);
> + }
>
> The "else" is not really necessary here because the "if" portion ends
> with "return
Fixed accordingly.
> One other point to note here is that if, by some chance, a failure
occurs as a result of receiving a memory-context interrupt, it's going
to fail the currently-running transaction
In the updated version, we make sure that no errors are being thrown
by ProcessMemoryContextInterrupt and other functions added by this
patch.
I have also added comments above the GetNamedDSA and GetNamedDsHash
functions to highlight the importance of minimizing errors in these
functions,
as they are called from CFI.
This way we try to prevent any non-critical errors from being thrown.
Following are responses to Daniel's comments:
> > Attached is the updated patch that addresses this.
>
> A few small comments on v47:
>
> +static const char *ContextTypeToString(NodeTag type);
> I think context_type_to_string() would be a better name on this internal
> function to model it closer to the existing int_list_to_array().
> Personally I
> would also place it before its first use to avoid the prototype, but that's
> personal preference.
>
>
Changed accordingly, also moved compute_context_path() so it appears
before its use to avoid the prototype.
>
> +static void
> +memstats_dsa_cleanup(char *key)
> This function warrants a documentation comment describing when it should be
> used safely.
>
Added a comment.
>
>
> + memstats_dsa_cleanup(key);
> + memstats_client_key_reset(procNumber);
> + ConditionVariableCancelSleep();
> + PG_RETURN_NULL();
> I think we should notify the user in these two timeout cases, why not
> adding an
> ereport(NOTICE, "request for memory context statistics timed out")); or
> something with a better wording than that.
>
> Added this.
> + Size sz = 0;
> + Size TotalProcs = 0;
> +
> + TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
> + TotalProcs = add_size(TotalProcs, MaxBackends);
> + sz = add_size(sz, mul_size(TotalProcs, sizeof(int)));
> +
> + return sz
> As we discussed off-list, the call to add_size() call can be omitted as it
> won't affect the calculation.
>
Fixed accordingly.
>
>
> +# Copyright (c) 2025, PostgreSQL Global Development Group
> Here, and possibly elsewhere, it should say 2026 instead I think.
>
>
> Fixed.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v49-0001-Add-function-to-report-memory-context-statistics.patch (59.9K, ../../CAH2L28umvzm9YwQBa_zrJM0ABC57ZAKPBxi8bD+n=1ADJ3SOZw@mail.gmail.com/3-v49-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From edf9562511c84984ac623733f18eda08ce6b0a68 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 159 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/dsm_registry.c | 8 +-
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1020 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 8 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
26 files changed, 1313 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 3ac81905d1f..af87b5b19ab 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,132 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5 seconds,
+ function returns NULL.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +428,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index eb9e31ae1bf..d1416e5534d 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -657,6 +657,17 @@ LANGUAGE INTERNAL
STRICT VOLATILE PARALLEL UNSAFE
AS 'pg_replication_origin_session_setup';
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -782,6 +793,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -808,6 +820,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 22379de1e31..a00bac51264 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -788,6 +788,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index 6482c21b8f9..1b7d5e7ffdc 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -685,6 +685,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index a2c0ff012c5..cb18ce80893 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 1a20387c4bd..3ac0fba225a 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -871,6 +871,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index a1a4f65f9a9..18a9e7f85e1 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index c3d56c866d3..49dc3e88023 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -879,6 +879,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/dsm_registry.c b/src/backend/storage/ipc/dsm_registry.c
index 068c1577b12..4810babac1b 100644
--- a/src/backend/storage/ipc/dsm_registry.c
+++ b/src/backend/storage/ipc/dsm_registry.c
@@ -270,7 +270,9 @@ GetNamedDSMSegment(const char *name, size_t size,
* This routine returns a pointer to the DSA. A new LWLock tranche ID will be
* generated if needed. Note that the lock tranche will be registered with the
* provided name. Also note that this should be called at most once for a
- * given DSA in each backend.
+ * given DSA in each backend. It's best to minimize the number of errors thrown
+ * in this function since it's called from a CFI function. An error here could
+ * lead to error in any transaction that calls the CFI function.
*/
dsa_area *
GetNamedDSA(const char *name, bool *found)
@@ -351,7 +353,9 @@ GetNamedDSA(const char *name, bool *found)
* params is ignored; a new LWLock tranche ID will be generated if needed.
* Note that the lock tranche will be registered with the provided name. Also
* note that this should be called at most once for a given table in each
- * backend.
+ * backend. It's best to minimize the number of errors thrown in this function
+ * since it's called from a CFI function. An error here could lead to error in
+ * any transaction that calls the CFI function.
*/
dshash_table *
GetNamedDSHash(const char *name, const dshash_parameters *params, bool *found)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 1f7e933d500..63159712003 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -141,6 +142,7 @@ CalculateShmemSize(void)
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
size = add_size(size, LogicalDecodingCtlShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize() + sizeof(LWLockPadded));
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
AioShmemInit();
WaitLSNShmemInit();
LogicalDecodingCtlShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 8e56922dcea..601e01d2574 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -694,6 +694,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 696bbb7b911..669ea2bfe58 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index b4a8d2f3a1c..d36bd2bd664 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3539,6 +3539,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index efde48e76b7..f3e541dcbb5 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -163,6 +163,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -364,6 +365,7 @@ SerialControl "Waiting to read or update shared <filename>pg_serial</filename> s
AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
WaitLSN "Waiting to read or update shared Wait-for-LSN state."
LogicalDecodingControl "Waiting to read or update logical decoding status information."
+MemContextReportingClientKeys "Waiting for another process to complete reading or writing the memory reporting keys."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 12b8d4cefaf..832b17fdf57 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,24 +15,128 @@
#include "postgres.h"
+#include "access/xact.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static void memstats_client_key_reset(int ProcNumber);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -62,6 +166,67 @@ int_list_to_array(const List *list)
return PointerGetDatum(result_array);
}
+/*
+ * context_type_to_string
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+static const char *
+context_type_to_string(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ {
+ elog(NOTICE, "hash table corrupted, can't construct path value");
+ return NIL;
+ }
+ path = lcons_int(cur_entry->context_id, path);
+ }
+ return path;
+}
+
/*
* PutMemoryContextsStatsTupleStore
* Add details for the given MemoryContext to 'tupstore'.
@@ -143,24 +308,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = context_type_to_string(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -305,3 +453,835 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Check if the server process slot is not empty and exit early Non-empty
+ * slot means some other client backend is requesting the statistics from
+ * the same server process.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] != -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(context_type_to_string(memcxt_info[i].type));
+
+ if (memcxt_info[i].levels != 0)
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+ else
+ nulls[3] = true;
+
+ if (memcxt_info[i].path[0] != 0)
+ {
+ int path_length;
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+/*
+ * This function is executed before returning from the
+ * pg_get_process_memory_contexts() function.
+ * It releases the DSA memory allocated for storing the memory statistics
+ * retrieved by that function. The caller must ensure that the LWLock for
+ * MemoryStatsDsHash is not already held.
+*/
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize(), &found);
+
+ if (!found)
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+
+ return mul_size(TotalProcs, sizeof(int));
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Avoid performing any shared memory operations in aborted transaction,
+ * the caller will timeout and return NULL.
+ */
+ if (IsAbortedTransactionBlockState())
+ return;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_SHARED);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ return;
+ }
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table. Here we just attach to those.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ INJECTION_POINT("memcontext-server-injection", NULL);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking
+ * required lock or this function may end up writing to unallocated
+ * memory.
+ */
+ if (!found || entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].levels = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextReset(memstats_ctx);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..e246fc7ce1f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..7e6de81e7d6 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -661,6 +661,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 073bdb35d2a..fd75345ff5f 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1011,6 +1011,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 5e5e33f64fc..b8a70ddb144 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8633,6 +8633,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..7e301affccc 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index e94ebce95b9..6dd80689bbb 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -87,6 +87,7 @@ PG_LWLOCK(52, SerialControl)
PG_LWLOCK(53, AioWorkerSubmissionQueue)
PG_LWLOCK(54, WaitLSN)
PG_LWLOCK(55, LogicalDecodingControl)
+PG_LWLOCK(56, MemContextReportingClientKeys)
/*
* There also exist several built-in LWLock tranches. As with the predefined
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index e52b8eb7697..4f1bcbf709f 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 11ab1717a16..6e8920bf28d 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -19,7 +19,6 @@
#include "nodes/memnodes.h"
-
/*
* MaxAllocSize, MaxAllocHugeSize
* Quasi-arbitrary limits on size of allocations.
@@ -319,4 +318,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3dd63fd88ed..c84e5adce16 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 004f9a70e00..d34d2b6211c 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 9f5ee8fd482..b83336f94ce 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1706,6 +1706,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] v49-0002-Test-module-to-test-memory-context-reporting-wit.patch (8.8K, ../../CAH2L28umvzm9YwQBa_zrJM0ABC57ZAKPBxi8bD+n=1ADJ3SOZw@mail.gmail.com/4-v49-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From 57127653459b7f2759f2ec30f566a1378dcb43ca Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 29 ++++
.../t/001_memcontext_inj.pl | 150 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 4 +
5 files changed, 196 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 44c7163c1cd..8ce9598495c 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..0a2dfc44f1c
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,29 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..e054ac29f05
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,150 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+#Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+#Server should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning for
+# one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+#Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+ok($psql_out = 'NULL');
+#Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+#Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+# Test if the monitoring works fine, when the client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+#Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+#Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name = 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-02-10 00:03 Rahila Syed <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2026-02-10 00:03 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: torikoshia <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
Hi,
Please find attached rebased and updated patches that include a few fixes
suggested by Daniel.
Fixes include
1. Changed LW_SHARED to LW_EXCLUSIVE when resetting client_keys.
2. Replaced calls to MemoryContextReset with MemoryContextDelete since a
new memory context is created each time.
3. Removed leftover code from earlier versions in ipci.c
4. Fix the TAP test to compare strings correctly.
5. Added more comments in the TAP test.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v50-0001-Add-function-to-report-memory-context-statistics.patch (59.9K, ../../CAH2L28vjHHdpoDKfcLT7nt765k_1X=R4qK7etQyf_DDUg-kNoQ@mail.gmail.com/3-v50-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From 64d533ef0719cc24fdedc1bc787ff9f8975c61bc Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 159 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/dsm_registry.c | 8 +-
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/storage/lmgr/proc.c | 1 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1020 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 8 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
26 files changed, 1313 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 3ac81905d1f..af87b5b19ab 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,132 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5 seconds,
+ function returns NULL.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +428,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index eb9e31ae1bf..d1416e5534d 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -657,6 +657,17 @@ LANGUAGE INTERNAL
STRICT VOLATILE PARALLEL UNSAFE
AS 'pg_replication_origin_session_setup';
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -782,6 +793,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -808,6 +820,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 6fde740465f..3c24119c43a 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -787,6 +787,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..d54193f8890 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -684,6 +684,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index a2c0ff012c5..cb18ce80893 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 82731e452fc..e13a40f1427 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -870,6 +870,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..1d6d8f22d20 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 2d8f57099fd..72aaf8ae5cc 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -878,6 +878,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/dsm_registry.c b/src/backend/storage/ipc/dsm_registry.c
index 068c1577b12..4810babac1b 100644
--- a/src/backend/storage/ipc/dsm_registry.c
+++ b/src/backend/storage/ipc/dsm_registry.c
@@ -270,7 +270,9 @@ GetNamedDSMSegment(const char *name, size_t size,
* This routine returns a pointer to the DSA. A new LWLock tranche ID will be
* generated if needed. Note that the lock tranche will be registered with the
* provided name. Also note that this should be called at most once for a
- * given DSA in each backend.
+ * given DSA in each backend. It's best to minimize the number of errors thrown
+ * in this function since it's called from a CFI function. An error here could
+ * lead to error in any transaction that calls the CFI function.
*/
dsa_area *
GetNamedDSA(const char *name, bool *found)
@@ -351,7 +353,9 @@ GetNamedDSA(const char *name, bool *found)
* params is ignored; a new LWLock tranche ID will be generated if needed.
* Note that the lock tranche will be registered with the provided name. Also
* note that this should be called at most once for a given table in each
- * backend.
+ * backend. It's best to minimize the number of errors thrown in this function
+ * since it's called from a CFI function. An error here could lead to error in
+ * any transaction that calls the CFI function.
*/
dshash_table *
GetNamedDSHash(const char *name, const dshash_parameters *params, bool *found)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 1f7e933d500..b0176613987 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -141,6 +142,7 @@ CalculateShmemSize(void)
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
size = add_size(size, LogicalDecodingCtlShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
AioShmemInit();
WaitLSNShmemInit();
LogicalDecodingCtlShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 8e56922dcea..601e01d2574 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -694,6 +694,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 8560a903bc8..f68583aa820 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 02e9aaa6bca..1478bce89a2 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3518,6 +3518,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 4aa864fe3c3..3e1dc4225b2 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -163,6 +163,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -366,6 +367,7 @@ SerialControl "Waiting to read or update shared <filename>pg_serial</filename> s
AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
WaitLSN "Waiting to read or update shared Wait-for-LSN state."
LogicalDecodingControl "Waiting to read or update logical decoding status information."
+MemContextReportingClientKeys "Waiting for another process to complete reading or writing the memory reporting keys."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index 12b8d4cefaf..7386e9527ae 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,24 +15,128 @@
#include "postgres.h"
+#include "access/xact.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(char *key);
+static void memstats_client_key_reset(int ProcNumber);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+#define MAX_PATH_DISPLAY_LENGTH 100
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -62,6 +166,67 @@ int_list_to_array(const List *list)
return PointerGetDatum(result_array);
}
+/*
+ * context_type_to_string
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+static const char *
+context_type_to_string(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ {
+ elog(NOTICE, "hash table corrupted, can't construct path value");
+ return NIL;
+ }
+ path = lcons_int(cur_entry->context_id, path);
+ }
+ return path;
+}
+
/*
* PutMemoryContextsStatsTupleStore
* Add details for the given MemoryContext to 'tupstore'.
@@ -143,24 +308,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = context_type_to_string(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -305,3 +453,835 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber procNumber;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ char key[CLIENT_KEY_SIZE];
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ procNumber = GetNumberFromPGProc(proc);
+
+ /*
+ * Check if the server process slot is not empty and exit early Non-empty
+ * slot means some other client backend is requesting the statistics from
+ * the same server process.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] != -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, sizeof(key), "%d", MyProcNumber);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[procNumber] == -1)
+ client_keys[procNumber] = MyProcNumber;
+ else
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, procNumber) < 0)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(context_type_to_string(memcxt_info[i].type));
+
+ if (memcxt_info[i].levels != 0)
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+ else
+ nulls[3] = true;
+
+ if (memcxt_info[i].path[0] != 0)
+ {
+ int path_length;
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(key);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+/*
+ * This function is executed before returning from the
+ * pg_get_process_memory_contexts() function.
+ * It releases the DSA memory allocated for storing the memory statistics
+ * retrieved by that function. The caller must ensure that the LWLock for
+ * MemoryStatsDsHash is not already held.
+*/
+static void
+memstats_dsa_cleanup(char *key)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, key, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize(), &found);
+
+ if (!found)
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+
+ return mul_size(TotalProcs, sizeof(int));
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Avoid performing any shared memory operations in aborted transaction,
+ * the caller will timeout and return NULL.
+ */
+ if (IsAbortedTransactionBlockState())
+ return;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ return;
+ }
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table. Here we just attach to those.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ snprintf(key, CLIENT_KEY_SIZE, "%d", clientProcNumber);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ INJECTION_POINT("memcontext-server-injection", NULL);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking
+ * required lock or this function may end up writing to unallocated
+ * memory.
+ */
+ if (!found || entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].levels = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ char key[CLIENT_KEY_SIZE];
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ snprintf(key, CLIENT_KEY_SIZE, "%d", idx);
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..e246fc7ce1f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index 3f401faf3de..7e6de81e7d6 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -661,6 +661,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 073bdb35d2a..fd75345ff5f 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1011,6 +1011,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 83f6501df38..de266a15570 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8633,6 +8633,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index db559b39c4d..7e301affccc 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index e94ebce95b9..6dd80689bbb 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -87,6 +87,7 @@ PG_LWLOCK(52, SerialControl)
PG_LWLOCK(53, AioWorkerSubmissionQueue)
PG_LWLOCK(54, WaitLSN)
PG_LWLOCK(55, LogicalDecodingControl)
+PG_LWLOCK(56, MemContextReportingClientKeys)
/*
* There also exist several built-in LWLock tranches. As with the predefined
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index e52b8eb7697..4f1bcbf709f 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
/* Recovery conflict reasons */
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 11ab1717a16..6e8920bf28d 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -19,7 +19,6 @@
#include "nodes/memnodes.h"
-
/*
* MaxAllocSize, MaxAllocHugeSize
* Quasi-arbitrary limits on size of allocations.
@@ -319,4 +318,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3dd63fd88ed..c84e5adce16 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 004f9a70e00..d34d2b6211c 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 7619845fba9..ee54eec00a5 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1707,6 +1707,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] v50-0002-Test-module-to-test-memory-context-reporting-wit.patch (9.0K, ../../CAH2L28vjHHdpoDKfcLT7nt765k_1X=R4qK7etQyf_DDUg-kNoQ@mail.gmail.com/4-v50-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From d694e726a06fcd4e934ad1dc3bb289f0d16fd904 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 29 ++++
.../t/001_memcontext_inj.pl | 154 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 4 +
5 files changed, 200 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 44c7163c1cd..8ce9598495c 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..0a2dfc44f1c
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,29 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..73665abf902
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,154 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+#Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+#Server should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning for
+# one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+
+# We use query_until so that we can execute additional commands without having to
+# wait for this command to complete, since it will pause at injection points.
+
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+#Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+is($psql_out, '', 'Query returns null');
+#Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+#Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test if the monitoring works fine, when the client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+#Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+#Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-02-10 04:50 Chao Li <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 2 replies; 65+ messages in thread
From: Chao Li @ 2026-02-10 04:50 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
> On Feb 10, 2026, at 08:03, Rahila Syed <[email protected]> wrote:
>
> Hi,
>
> Please find attached rebased and updated patches that include a few fixes
> suggested by Daniel.
>
> Fixes include
> 1. Changed LW_SHARED to LW_EXCLUSIVE when resetting client_keys.
> 2. Replaced calls to MemoryContextReset with MemoryContextDelete since a
> new memory context is created each time.
> 3. Removed leftover code from earlier versions in ipci.c
> 4. Fix the TAP test to compare strings correctly.
> 5. Added more comments in the TAP test.
>
> Thank you,
> Rahila Syed
> <v50-0001-Add-function-to-report-memory-context-statistics.patch><v50-0002-Test-module-to-test-memory-context-reporting-wit.patch>
Hi Rahila,
Thanks for the patch. I applied v50 locally and played with it a little bit then I reviewed v50. Basically I think it’s convenient to query a server process’ memory usage via a SQL statement.
Here comes my review comments.
1 - 0001- mcxt.c
```
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
```
As the header comment says that “totals should not be NULL”, maybe add Assert(total!=NULL) to ensure that.
2 - 0001- mcxt.c
```
+ *num_contexts = 0;
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = *num_contexts + ichild;
+}
```
*num_contexts is only initialized to 0, then *num_contexts = *num_contexts + ichild. Looks like the the initialization is unnecessary, and the assignment can just be *num_contexts = ichild.
3 - 0001 - mcxtfuncs.c
```
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[100];
```
You already defined a constant MAX_PATH_DISPLAY_LENGTH below, would it make sense to pull up the macro definition and use it for “path” definition?
4 - 0001 - mctxfuncs.c
```
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ char key[64];
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_strcmp,
+ dshash_strhash,
+ dshash_strcpy
+};
```
I wonder why we cannot just use the integer ProcNumber as the hash key? I think dshash fully supports fixed-size binary keys. We can use dshash_memcmp and dshash_memhash for compare and hash functions.
5 - 0001 - mctxfuncs.c
```
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ {
+ elog(NOTICE, "hash table corrupted, can't construct path value");
+ return NIL;
+ }
+ path = lcons_int(cur_entry->context_id, path);
+ }
+ return path;
+}
```
As in the hash entry, path is a 100-elements array, would it make sense to also limit the path list to not append more than 100 items in this function?
6 - 0001 - mctxfuncs.c
```
+ entry->stats_written = true;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+}
```
This looks like a race condition. ConditionVariableSignal(&entry->memcxt_cv); is called after the lock is released. So, there is a chance that a process has been terminated, and its before_shmem_exit callback acquired the entry lock and deleted the entry from the hash. So, I think we should send the signal before releasing the lock.
7 - 0001 - mctxfuncs.c
```
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
```
For the wait loop, if the condition wakes up for some other reason, it will loop back and wait for the other 5 seconds, then total wait period will exceed 5 seconds, maybe 9 seconds. This is not a big deal, but the doc explicitly says “If the process does not respond with memory contexts statistics in 5 seconds”, so that behavior might be inconsistent with the doc.
I think we can calculate remaining time and pass remaining milliseconds into ConditionVariableTimedSleep.
8 - 0001 - mctxfuncs.c
```
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(key);
+ memstats_client_key_reset(procNumber);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, key, &found);
+ Assert(found);
```
After ConditionVariableTimedSleep, rather than Assert(found), I think we should check if (!found). Because it waits for 5 seconds without holding the lock, the process may terminate and delete the entry during the period.
9 - 0001 - proc.c
```
diff --git a/src/backend/storage/lmgr/proc.c b/src/backend/storage/lmgr/proc.c
index 8560a903bc8..f68583aa820 100644
--- a/src/backend/storage/lmgr/proc.c
+++ b/src/backend/storage/lmgr/proc.c
@@ -51,6 +51,7 @@
#include "storage/procsignal.h"
#include "storage/spin.h"
#include "storage/standby.h"
+#include "utils/memutils.h"
#include "utils/timeout.h"
#include "utils/timestamp.h"
```
This file is sololy added an include without any other change, that seems unneeded. I tried to remove this include, and the build still passed.
Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-02-15 12:40 Rahila Syed <[email protected]>
parent: Chao Li <[email protected]>
1 sibling, 0 replies; 65+ messages in thread
From: Rahila Syed @ 2026-02-15 12:40 UTC (permalink / raw)
To: Chao Li <[email protected]>; +Cc: Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
Hi Chao,
Thank you for testing and reviewing. Please find attached the updated
patches
that incorporate your review comments.
>
> As the header comment says that “totals should not be NULL”, maybe add
> Assert(total!=NULL) to ensure that.
>
>
Added this.
> + /*
> + * Add the count of all the children contexts which are traversed
> + * including the parent.
> + */
> + *num_contexts = *num_contexts + ichild;
> +}
> ```
>
> *num_contexts is only initialized to 0, then *num_contexts = *num_contexts
> + ichild. Looks like the the initialization is unnecessary, and the
> assignment can just be *num_contexts = ichild.
>
Fixed accordingly.
>
> + int path[100];
> ```
>
> You already defined a constant MAX_PATH_DISPLAY_LENGTH below, would it
> make sense to pull up the macro definition and use it for “path” definition?
>
>
Makes sense, done in the attached patch.
> +
> +static const dshash_parameters memctx_dsh_params = {
> + offsetof(MemoryStatsDSHashEntry, memcxt_cv),
> + sizeof(MemoryStatsDSHashEntry),
> + dshash_strcmp,
> + dshash_strhash,
> + dshash_strcpy
> +};
> ```
>
> I wonder why we cannot just use the integer ProcNumber as the hash key? I
> think dshash fully supports fixed-size binary keys. We can use
> dshash_memcmp and dshash_memhash for compare and hash functions.
>
I have incorporated your suggestion in the updated patch—using integer
keys will be more efficient
for memory usage and will help avoid conversions between data types.
Thank you for the suggestion.
+static List *
> +compute_context_path(MemoryContext c, HTAB *context_id_lookup)
> +{
> As in the hash entry, path is a 100-elements array, would it make sense to
> also limit the path list to not append more than 100 items in this function?
>
We need to calculate the full path list to report the number of levels in
the tree.
We could choose to return the number of levels separately from the path
list and limit the
path list to no more than 100 levels, but we would still have to iterate
through all the levels.
Returning a single variable that captures both levels and path keeps the
function signature simple.
Kindly let me know your view.
> + dshash_release_lock(MemoryStatsDsHash, entry);
> + hash_destroy(context_id_lookup);
> +
> + MemoryContextSwitchTo(oldcontext);
> + MemoryContextDelete(memstats_ctx);
> + /* Notify waiting client backend and return */
> + ConditionVariableSignal(&entry->memcxt_cv);
> +}
> ```
>
> This looks like a race condition.
> ConditionVariableSignal(&entry->memcxt_cv); is called after the lock is
> released. So, there is a chance that a process has been terminated, and its
> before_shmem_exit callback acquired the entry lock and deleted the entry
> from the hash. So, I think we should send the signal before releasing the
> lock.
>
>
Good catch. Fixed accordingly.
>
> + */
> + if (ConditionVariableTimedSleep(&entry->memcxt_cv,
> +
> (MEMORY_STATS_MAX_TIMEOUT * 1000),
> +
> WAIT_EVENT_MEM_CXT_PUBLISH))
> ```
>
> For the wait loop, if the condition wakes up for some other reason, it
> will loop back and wait for the other 5 seconds, then total wait period
> will exceed 5 seconds, maybe 9 seconds. This is not a big deal, but the doc
> explicitly says “If the process does not respond with memory contexts
> statistics in 5 seconds”, so that behavior might be inconsistent with the
> doc.
>
> I think we can calculate remaining time and pass remaining milliseconds
> into ConditionVariableTimedSleep.
>
>
The following check in the beginning of the wait loop is intended to avoid
this.
/* Return if we have already exceeded the timeout */
if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
{
memstats_dsa_cleanup(client_procno);
memstats_client_key_reset(server_procno);
ConditionVariableCancelSleep();
ereport(NOTICE,
errmsg("request for memory context statistics for PID
%d timed out",
pid));
PG_RETURN_NULL();
}
It may still exceed by an amount less than 5 seconds, but it will always be
less
than 2 * MEMORY_STATS_MAX_TIMEOUT, because
we track the elapsed time and exit after the condition above is satisfied.
Maybe we should edit the docs to reflect this.
After ConditionVariableTimedSleep, rather than Assert(found), I think we
> should check if (!found). Because it waits for 5 seconds without holding
> the lock, the process may terminate and delete the entry during the period.
>
>
The client backend deletes the entry only upon exiting; no other process
can remove the entry.
Therefore, it is not possible for the client backend to execute this Assert
after the entry has been
deleted.
>
> This file is sololy added an include without any other change, that seems
> unneeded. I tried to remove this include, and the build still passed.
>
>
Fixed. Thank you for noticing this. It looks like a remnant from earlier
versions.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v51-0001-Add-function-to-report-memory-context-statistics.patch (59.5K, ../../CAH2L28sC0Bo5nKzmzUaB4_bg1gVM=TSvD18MjkPQ2tkbjc0vjg@mail.gmail.com/3-v51-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From ee89c9f1283475aae4b403246aedf7abd182951f Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 159 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/dsm_registry.c | 8 +-
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1013 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 8 +-
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
25 files changed, 1305 insertions(+), 23 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 3ac81905d1f..af87b5b19ab 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,132 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5 seconds,
+ function returns NULL.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, which each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. When
+ <literal>num_agg_contexts</literal> is <literal>1</literal> it means
+ that the context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +428,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index eb9e31ae1bf..d1416e5534d 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -657,6 +657,17 @@ LANGUAGE INTERNAL
STRICT VOLATILE PARALLEL UNSAFE
AS 'pg_replication_origin_session_setup';
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -782,6 +793,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -808,6 +820,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 6fde740465f..3c24119c43a 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -787,6 +787,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..d54193f8890 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -684,6 +684,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index a2c0ff012c5..cb18ce80893 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 82731e452fc..e13a40f1427 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -870,6 +870,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..1d6d8f22d20 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 2d8f57099fd..72aaf8ae5cc 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -878,6 +878,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/dsm_registry.c b/src/backend/storage/ipc/dsm_registry.c
index 068c1577b12..4810babac1b 100644
--- a/src/backend/storage/ipc/dsm_registry.c
+++ b/src/backend/storage/ipc/dsm_registry.c
@@ -270,7 +270,9 @@ GetNamedDSMSegment(const char *name, size_t size,
* This routine returns a pointer to the DSA. A new LWLock tranche ID will be
* generated if needed. Note that the lock tranche will be registered with the
* provided name. Also note that this should be called at most once for a
- * given DSA in each backend.
+ * given DSA in each backend. It's best to minimize the number of errors thrown
+ * in this function since it's called from a CFI function. An error here could
+ * lead to error in any transaction that calls the CFI function.
*/
dsa_area *
GetNamedDSA(const char *name, bool *found)
@@ -351,7 +353,9 @@ GetNamedDSA(const char *name, bool *found)
* params is ignored; a new LWLock tranche ID will be generated if needed.
* Note that the lock tranche will be registered with the provided name. Also
* note that this should be called at most once for a given table in each
- * backend.
+ * backend. It's best to minimize the number of errors thrown in this function
+ * since it's called from a CFI function. An error here could lead to error in
+ * any transaction that calls the CFI function.
*/
dshash_table *
GetNamedDSHash(const char *name, const dshash_parameters *params, bool *found)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 1f7e933d500..b0176613987 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -141,6 +142,7 @@ CalculateShmemSize(void)
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
size = add_size(size, LogicalDecodingCtlShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
AioShmemInit();
WaitLSNShmemInit();
LogicalDecodingCtlShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 5d33559926a..c688f3d36aa 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -694,6 +694,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 21de158adbb..c4b009fa532 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3573,6 +3573,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 4aa864fe3c3..3e1dc4225b2 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -163,6 +163,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -366,6 +367,7 @@ SerialControl "Waiting to read or update shared <filename>pg_serial</filename> s
AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
WaitLSN "Waiting to read or update shared Wait-for-LSN state."
LogicalDecodingControl "Waiting to read or update logical decoding status information."
+MemContextReportingClientKeys "Waiting for another process to complete reading or writing the memory reporting keys."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c7f7b8bc2dd..72f25dad3f3 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,25 +15,129 @@
#include "postgres.h"
+#include "access/xact.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "miscadmin.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "storage/procsignal.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+#define MAX_PATH_DISPLAY_LENGTH 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[MAX_PATH_DISPLAY_LENGTH];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ ProcNumber client_key;
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_memcmp,
+ dshash_memhash,
+ dshash_memcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(int ProcNumber);
+static void memstats_client_key_reset(int ProcNumber);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -63,6 +167,67 @@ int_list_to_array(const List *list)
return PointerGetDatum(result_array);
}
+/*
+ * context_type_to_string
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+static const char *
+context_type_to_string(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ {
+ elog(NOTICE, "hash table corrupted, can't construct path value");
+ return NIL;
+ }
+ path = lcons_int(cur_entry->context_id, path);
+ }
+ return path;
+}
+
/*
* PutMemoryContextsStatsTupleStore
* Add details for the given MemoryContext to 'tupstore'.
@@ -144,24 +309,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = context_type_to_string(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -306,3 +454,828 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber client_procno = MyProcNumber;
+ ProcNumber server_procno;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ server_procno = GetNumberFromPGProc(proc);
+
+ /*
+ * Check if the server process slot is not empty and exit early Non-empty
+ * slot means some other client backend is requesting the statistics from
+ * the same server process.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[server_procno] != -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &client_procno, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[server_procno] == -1)
+ client_keys[server_procno] = MyProcNumber;
+ else
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, server_procno) < 0)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for MEMORY_STATS_MAX_TIMEOUT. If no statistics are available
+ * within the allowed time then return NULL. The timer is defined in
+ * milliseconds since that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000),
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &client_procno, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(context_type_to_string(memcxt_info[i].type));
+
+ if (memcxt_info[i].levels != 0)
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+ else
+ nulls[3] = true;
+
+ if (memcxt_info[i].path[0] != 0)
+ {
+ int path_length;
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(client_procno);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+/*
+ * This function is executed before returning from the
+ * pg_get_process_memory_contexts() function.
+ * It releases the DSA memory allocated for storing the memory statistics
+ * retrieved by that function. The caller must ensure that the LWLock for
+ * MemoryStatsDsHash is not already held.
+*/
+static void
+memstats_dsa_cleanup(int procNumber)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, &procNumber, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize(), &found);
+
+ if (!found)
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+
+ return mul_size(TotalProcs, sizeof(int));
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Avoid performing any shared memory operations in aborted transaction,
+ * the caller will timeout and return NULL.
+ */
+ if (IsAbortedTransactionBlockState())
+ return;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ return;
+ }
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table. Here we just attach to those.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &clientProcNumber, &found);
+ INJECTION_POINT("memcontext-server-injection", NULL);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking
+ * required lock or this function may end up writing to unallocated
+ * memory.
+ */
+ if (!found || entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].levels = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(context->ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &idx, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..e246fc7ce1f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b59e08605cc..ba5d3bf0d7a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -669,6 +669,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 073bdb35d2a..2a826cbef17 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1011,6 +1011,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ Assert(totals != NULL);
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 83f6501df38..de266a15570 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8633,6 +8633,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified backend',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'r',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9..38e1e55a9a6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index e94ebce95b9..6dd80689bbb 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -87,6 +87,7 @@ PG_LWLOCK(52, SerialControl)
PG_LWLOCK(53, AioWorkerSubmissionQueue)
PG_LWLOCK(54, WaitLSN)
PG_LWLOCK(55, LogicalDecodingControl)
+PG_LWLOCK(56, MemContextReportingClientKeys)
/*
* There also exist several built-in LWLock tranches. As with the predefined
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 348fba53a93..00ccfcb22a4 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
PROCSIG_RECOVERY_CONFLICT, /* backend is blocking recovery, check
* PGPROC->pendingRecoveryConflicts for the
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 11ab1717a16..6e8920bf28d 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -19,7 +19,6 @@
#include "nodes/memnodes.h"
-
/*
* MaxAllocSize, MaxAllocHugeSize
* Quasi-arbitrary limits on size of allocations.
@@ -319,4 +318,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3dd63fd88ed..c84e5adce16 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 004f9a70e00..d34d2b6211c 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..f90ff78a78b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1709,6 +1709,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] v51-0002-Test-module-to-test-memory-context-reporting-wit.patch (9.0K, ../../CAH2L28sC0Bo5nKzmzUaB4_bg1gVM=TSvD18MjkPQ2tkbjc0vjg@mail.gmail.com/4-v51-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From f9467a7a49f884b90e61c339d3904cb0ad1a6d7a Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 29 ++++
.../t/001_memcontext_inj.pl | 154 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 4 +
5 files changed, 200 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 44c7163c1cd..8ce9598495c 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..0a2dfc44f1c
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,29 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..73665abf902
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,154 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+#Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+#Server should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning for
+# one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+
+# We use query_until so that we can execute additional commands without having to
+# wait for this command to complete, since it will pause at injection points.
+
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+#Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+is($psql_out, '', 'Query returns null');
+#Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+#Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test if the monitoring works fine, when the client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+#Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+#Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-02-20 23:13 Rahila Syed <[email protected]>
parent: Chao Li <[email protected]>
1 sibling, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2026-02-20 23:13 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Daniel Gustafsson <[email protected]>; torikoshia <[email protected]>; Robert Haas <[email protected]>; Chao Li <[email protected]>
Hi,
Please find attached updated and rebased patches which incorporate
code fixes suggested off-list by Daniel
These changes include a fix to free dsa memory in one of the
exit paths from the client-side function along with several documentation
and comment improvements.
> I think we can calculate remaining time and pass remaining milliseconds
> into ConditionVariableTimedSleep.
>
>
This comment by Chao Li was also addressed as part of Daniel's review
fixes.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v52-0001-Add-function-to-report-memory-context-statistics.patch (59.8K, ../../CAH2L28sZ5ELWzXgRNJEYpOR8jk6DdWETgJ7L-xYa9SXQ+n5Z7w@mail.gmail.com/3-v52-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From c56ac6ce0d9fcf456434fa2b8e50bd7aa7091d9f Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 159 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/dsm_registry.c | 8 +-
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1020 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 7 +
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
25 files changed, 1312 insertions(+), 22 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 3ac81905d1f..120f032f249 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,132 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to backends as
+ well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5 seconds,
+ the function returns NULL.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, with each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number of aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics. A
+ <literal>num_agg_contexts</literal> value of <literal>1</literal>
+ indicates that context statistics are displayed separately.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +428,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 69699f8830a..b62944cee87 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -378,6 +378,17 @@ BEGIN ATOMIC
END;
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -503,6 +514,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -529,6 +541,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 6fde740465f..3c24119c43a 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -787,6 +787,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..d54193f8890 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -684,6 +684,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index a2c0ff012c5..cb18ce80893 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 82731e452fc..e13a40f1427 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -870,6 +870,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..1d6d8f22d20 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 742137edad6..d5a3cb13a4c 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -875,6 +875,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/dsm_registry.c b/src/backend/storage/ipc/dsm_registry.c
index 068c1577b12..4810babac1b 100644
--- a/src/backend/storage/ipc/dsm_registry.c
+++ b/src/backend/storage/ipc/dsm_registry.c
@@ -270,7 +270,9 @@ GetNamedDSMSegment(const char *name, size_t size,
* This routine returns a pointer to the DSA. A new LWLock tranche ID will be
* generated if needed. Note that the lock tranche will be registered with the
* provided name. Also note that this should be called at most once for a
- * given DSA in each backend.
+ * given DSA in each backend. It's best to minimize the number of errors thrown
+ * in this function since it's called from a CFI function. An error here could
+ * lead to error in any transaction that calls the CFI function.
*/
dsa_area *
GetNamedDSA(const char *name, bool *found)
@@ -351,7 +353,9 @@ GetNamedDSA(const char *name, bool *found)
* params is ignored; a new LWLock tranche ID will be generated if needed.
* Note that the lock tranche will be registered with the provided name. Also
* note that this should be called at most once for a given table in each
- * backend.
+ * backend. It's best to minimize the number of errors thrown in this function
+ * since it's called from a CFI function. An error here could lead to error in
+ * any transaction that calls the CFI function.
*/
dshash_table *
GetNamedDSHash(const char *name, const dshash_parameters *params, bool *found)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 1f7e933d500..b0176613987 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -141,6 +142,7 @@ CalculateShmemSize(void)
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
size = add_size(size, LogicalDecodingCtlShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
AioShmemInit();
WaitLSNShmemInit();
LogicalDecodingCtlShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 7505c9d3a37..839cbdbdc89 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -695,6 +695,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index d01a09dd0c4..9e0d289a604 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3573,6 +3573,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 4aa864fe3c3..3e1dc4225b2 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -163,6 +163,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -366,6 +367,7 @@ SerialControl "Waiting to read or update shared <filename>pg_serial</filename> s
AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
WaitLSN "Waiting to read or update shared Wait-for-LSN state."
LogicalDecodingControl "Waiting to read or update logical decoding status information."
+MemContextReportingClientKeys "Waiting for another process to complete reading or writing the memory reporting keys."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c7f7b8bc2dd..f18b6d47c60 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,25 +15,128 @@
#include "postgres.h"
+#include "access/xact.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "storage/procsignal.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+#define MAX_PATH_DISPLAY_LENGTH 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[MAX_PATH_DISPLAY_LENGTH];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ ProcNumber client_key;
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_memcmp,
+ dshash_memhash,
+ dshash_memcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(int ProcNumber);
+static void memstats_client_key_reset(int ProcNumber);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -63,6 +166,67 @@ int_list_to_array(const List *list)
return PointerGetDatum(result_array);
}
+/*
+ * context_type_to_string
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+static const char *
+context_type_to_string(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ {
+ elog(NOTICE, "hash table corrupted, can't construct path value");
+ return NIL;
+ }
+ path = lcons_int(cur_entry->context_id, path);
+ }
+ return path;
+}
+
/*
* PutMemoryContextsStatsTupleStore
* Add details for the given MemoryContext to 'tupstore'.
@@ -144,24 +308,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = context_type_to_string(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -306,3 +453,836 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber client_procno = MyProcNumber;
+ ProcNumber server_procno;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ server_procno = GetNumberFromPGProc(proc);
+
+ /*
+ * First, check if the server process slot is occupied; if so, exit early.
+ * An occupied slot indicates another client backend is already requesting
+ * statistics from this server process. We should not set the slot yet,
+ * even if it appears empty, until we are prepared to signal the server
+ * process. This is because if the following DSA APIs throw an error, the
+ * server slot may not be reset, leading to it being incorrectly marked as
+ * occupied and blocking further memory context statistics queries to the
+ * same process.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[server_procno] != -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &client_procno, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[server_procno] == -1)
+ client_keys[server_procno] = MyProcNumber;
+ else
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ memstats_dsa_cleanup(client_procno);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, server_procno) < 0)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for at most MEMORY_STATS_MAX_TIMEOUT seconds, if we have been
+ * woken up spuriously we subtract the time already slept. If no
+ * statistics are available within the allowed time then return NULL.
+ * The timer is defined in milliseconds since that's what the
+ * condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000) - elapsed_time,
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &client_procno, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(context_type_to_string(memcxt_info[i].type));
+
+ if (memcxt_info[i].levels != 0)
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+ else
+ nulls[3] = true;
+
+ if (memcxt_info[i].path[0] != 0)
+ {
+ int path_length;
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(client_procno);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+/*
+ * This function is executed before returning from the
+ * pg_get_process_memory_contexts() function.
+ * It releases the DSA memory allocated for storing the memory statistics
+ * retrieved by that function. The caller must ensure that the LWLock for
+ * MemoryStatsDsHash is not already held.
+*/
+static void
+memstats_dsa_cleanup(int procNumber)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, &procNumber, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize(), &found);
+
+ if (!found)
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+
+ return mul_size(TotalProcs, sizeof(int));
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Avoid performing any shared memory operations in aborted transaction,
+ * the caller will timeout and return NULL.
+ */
+ if (IsAbortedTransactionBlockState())
+ return;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ return;
+ }
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table. Here we just attach to those.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &clientProcNumber, &found);
+ INJECTION_POINT("memcontext-server-injection", NULL);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking
+ * required lock or this function may end up writing to unallocated
+ * memory.
+ */
+ if (!found || entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContexts children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].levels = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &idx, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..e246fc7ce1f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b59e08605cc..ba5d3bf0d7a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -669,6 +669,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 073bdb35d2a..2a826cbef17 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1011,6 +1011,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ Assert(totals != NULL);
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index dac40992cbc..3667bc2bf41 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8641,6 +8641,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified process',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'u',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9..38e1e55a9a6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index e94ebce95b9..6dd80689bbb 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -87,6 +87,7 @@ PG_LWLOCK(52, SerialControl)
PG_LWLOCK(53, AioWorkerSubmissionQueue)
PG_LWLOCK(54, WaitLSN)
PG_LWLOCK(55, LogicalDecodingControl)
+PG_LWLOCK(56, MemContextReportingClientKeys)
/*
* There also exist several built-in LWLock tranches. As with the predefined
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 348fba53a93..00ccfcb22a4 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
PROCSIG_RECOVERY_CONFLICT, /* backend is blocking recovery, check
* PGPROC->pendingRecoveryConflicts for the
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 11ab1717a16..40099036308 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -319,4 +319,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3dd63fd88ed..c84e5adce16 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 004f9a70e00..d34d2b6211c 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..f90ff78a78b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1709,6 +1709,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] v52-0002-Test-module-to-test-memory-context-reporting-wit.patch (8.9K, ../../CAH2L28sZ5ELWzXgRNJEYpOR8jk6DdWETgJ7L-xYa9SXQ+n5Z7w@mail.gmail.com/4-v52-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From 705b52ec7d0c66352e0c31397ff5b31f1e018f0d Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 25 +++
.../t/001_memcontext_inj.pl | 154 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 4 +
5 files changed, 196 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 44c7163c1cd..8ce9598495c 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..47bce45bc1d
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,25 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+TAP_TESTS = 1
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+REGRESS = test_memcontext_reporting
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..73665abf902
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,154 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+#Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+#Server should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+#Query the same process after detaching the injection point, using some other client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning for
+# one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+
+# We use query_until so that we can execute additional commands without having to
+# wait for this command to complete, since it will pause at injection points.
+
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+#Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+is($psql_out, '', 'Query returns null');
+#Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+#Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test if the monitoring works fine, when the client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+#Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+#Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-02-23 09:58 Daniel Gustafsson <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Daniel Gustafsson @ 2026-02-23 09:58 UTC (permalink / raw)
To: Rahila Syed <[email protected]>; +Cc: pgsql-hackers; torikoshia <[email protected]>; Robert Haas <[email protected]>; Chao Li <[email protected]>
> On 21 Feb 2026, at 00:13, Rahila Syed <[email protected]> wrote:
> Please find attached updated and rebased patches which incorporate
> code fixes suggested off-list by Daniel
>
> These changes include a fix to free dsa memory in one of the
> exit paths from the client-side function along with several documentation
> and comment improvements.
Thanks for the new version, I think this is looking pretty good now. The
attached .diff.txt (renamed to keep the CFBot from grabbing it) has mostly
whitespace fixes, but also a few small things which are discussed below:
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
Re-reading the docs I think the para's are in the wrong order, as the sentence
above makes little sense to be after the output has already been discussed. I
think this needs to go earlier, see the attached .diff.txt for what I am
proposing.
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table. Here we just attach to those.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
I think this needs an expanded comment discussing why there is no need the
check the DSA and DShash after the GetNamed_ functions, since they can be NULL
going in to this. The attached .diff.txt has a proposal along with an
Assertion to keep future changes of the API from risking subtly breaking this
assumption.
+REGRESS = test_memcontext_reporting
Since the test module doesn't contain any pg_regress style .sql/.out tests so
this line in the Makefile cause a test failure when running with Autoconf.
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+max_connections = 100
+log_statement = none
+restart_after_crash = false
+]);
Why do we need to set max_connections for this test?
+#Server should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
This test doesn't validate that the server actually errored does it? (There is
no proposed fix in the attached.)
--
Daniel Gustafsson
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 120f032f249..aa83b11e131 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -274,10 +274,19 @@
<para>
This function handles requests to display the memory contexts of a
<productname>PostgreSQL</productname> process with the specified
- process ID. The function can be used to send requests to backends as
- well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
- If the process does not respond with memory contexts statistics in 5 seconds,
- the function returns NULL.
+ process ID. The function can be used to send requests to
+ <glossterm linkend="glossary-backend">backends</glossterm> as well as
+ <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5
+ seconds, the function returns NULL.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics.
</para>
<para>
The returned record contains extended statistics per each memory
@@ -350,7 +359,8 @@
<listitem>
<para>
<parameter>num_agg_contexts</parameter> - The number of memory
- contexts aggregated in the displayed statistics.
+ contexts aggregated in the displayed statistics. <literal>1</literal>
+ indicates that context statistics are displayed separately.
</para>
</listitem>
</itemizedlist>
@@ -365,16 +375,6 @@
<parameter>summary</parameter> is <literal>false</literal> (the default),
<literal>the num_agg_contexts</literal> value is <literal>1</literal>,
indicating that individual statistics are being displayed.
- </para>
- <para>
- After receiving memory context statistics from the target process, it
- returns the results as one row per context. If all the contexts don't
- fit within the pre-determined size limit, the remaining context
- statistics are aggregated and a cumulative total is displayed. The
- <literal>num_agg_contexts</literal> column indicates the number of
- contexts aggregated in the displayed statistics. A
- <literal>num_agg_contexts</literal> value of <literal>1</literal>
- indicates that context statistics are displayed separately.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index f18b6d47c60..fdf0043211b 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -978,7 +978,11 @@ ProcessGetMemoryContextInterrupt(void)
/*
* The client process should have created the required DSA and DSHash
- * table. Here we just attach to those.
+ * table, so we can connect to these directly. The GetNamedXXX functions
+ * either return DSA/DSHash or error out (which we want to avoid as that
+ * would cause the current transaction to fail) so we don't need to check
+ * for NULL. An assertion is enough to catch in case this API should be
+ * changed at some point.
*/
if (MemoryStatsDsaArea == NULL)
MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
@@ -988,6 +992,8 @@ ProcessGetMemoryContextInterrupt(void)
MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
&memctx_dsh_params, &found);
+ Assert(MemoryStatsDsaArea != NULL && MemoryStatsDsHash != NULL);
+
/*
* The entry lock is held by dshash_find_or_insert to protect writes to
* process specific memory. Two different processes publishing statistics
@@ -1001,9 +1007,8 @@ ProcessGetMemoryContextInterrupt(void)
* if the caller has timed out waiting for us and have issued a request to
* another backend.
*
- * Make sure that the client always deletes the entry after taking
- * required lock or this function may end up writing to unallocated
- * memory.
+ * Make sure that the client always deletes the entry after taking the
+ * required lock or this function may end up writing to unallocated memory.
*/
if (!found || entry->target_server_id != MyProcPid)
{
@@ -1042,13 +1047,11 @@ ProcessGetMemoryContextInterrupt(void)
HASH_ENTER, &found);
Assert(!found);
- /*
- * context id starts with 1
- */
+ /* context id starts with 1 */
contextid_entry->context_id = cxt_id + 1;
/*
- * Copy statistics for each of TopMemoryContexts children. This
+ * Copy statistics for each of TopMemoryContext's children. This
* includes statistics of at most 100 children per node, with each
* child node limited to a depth of 100 in its subtree.
*/
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
index 47bce45bc1d..2ce169ffa69 100644
--- a/src/test/modules/test_memcontext_reporting/Makefile
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -11,8 +11,6 @@ OBJS = \
test_memcontext_reporting.o
PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
-REGRESS = test_memcontext_reporting
-
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
index 73665abf902..152465c1570 100644
--- a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -21,7 +21,6 @@ $node->init;
$node->append_conf(
'postgresql.conf',
qq[
-max_connections = 100
log_statement = none
restart_after_crash = false
]);
@@ -36,7 +35,7 @@ $node->safe_psql('postgres',
my $pid = $node->safe_psql('postgres',
"SELECT pid from pg_stat_activity where backend_type='checkpointer'");
-#Client should have thrown error
+# Client should have thrown error
$node->psql(
'postgres',
qq(select pg_get_process_memory_contexts($pid, true);),
@@ -44,7 +43,8 @@ $node->psql(
like($psql_err,
qr/error triggered for injection point memcontext-client-injection/);
-#Query the same process after detaching the injection point, using some other client and it should succeed.
+# Query the same process after detaching the injection point, using some other
+# client and it should succeed.
$node->safe_psql('postgres',
"select injection_points_detach('memcontext-client-injection');");
my $topcontext_name = $node->safe_psql('postgres',
@@ -57,13 +57,14 @@ $node->safe_psql('postgres',
"select injection_points_attach('memcontext-server-injection', 'error');"
);
-#Server should have thrown error
+# Server should have thrown error
$node->psql(
'postgres',
qq(select pg_get_process_memory_contexts($pid, true);),
stderr => \$psql_err);
-#Query the same process after detaching the injection point, using some other client and it should succeed.
+# Query the same process after detaching the injection point, using some other
+# client and it should succeed.
$node->safe_psql('postgres',
"select injection_points_detach('memcontext-server-injection');");
$topcontext_name = $node->safe_psql('postgres',
@@ -71,8 +72,8 @@ $topcontext_name = $node->safe_psql('postgres',
);
ok($topcontext_name eq 'TopMemoryContext');
-# Test that two concurrent requests to the same process results in a warning for
-# one of those
+# Test that two concurrent requests to the same process results in a warning
+# for one of those
$node->safe_psql('postgres',
"SELECT injection_points_attach('memcontext-client-injection', 'wait');");
@@ -80,8 +81,9 @@ $node->safe_psql('postgres',
"SELECT injection_points_attach('memcontext-server-wait', 'wait');");
my $psql_session1 = $node->background_psql('postgres');
-# We use query_until so that we can execute additional commands without having to
-# wait for this command to complete, since it will pause at injection points.
+# We use query_until so that we can execute additional commands without having
+# to wait for this command to complete, since it will pause at injection
+# points.
$psql_session1->query_until(
qr//,
@@ -95,7 +97,7 @@ $node->psql(
stderr => \$psql_err);
ok($psql_err =~
/WARNING: server process $pid is processing previous request/);
-#Wake the client up.
+# Wake the client up.
$node->safe_psql('postgres',
"SELECT injection_points_wakeup('memcontext-client-injection');");
@@ -104,7 +106,8 @@ $node->safe_psql('postgres',
$node->safe_psql('postgres',
"select injection_points_detach('memcontext-server-wait');");
-# Test the client process exiting with timeout does not break the server process
+# Test the client process exiting with timeout does not break the server
+# process
$node->safe_psql('postgres',
"SELECT injection_points_attach('memcontext-server-wait', 'wait');");
@@ -114,12 +117,12 @@ $node->psql(
qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
stdout => \$psql_out);
is($psql_out, '', 'Query returns null');
-#Wakeup the server process up and detach the injection point.
+# Wakeup the server process up and detach the injection point.
$node->safe_psql('postgres',
"SELECT injection_points_wakeup('memcontext-server-wait');");
$node->safe_psql('postgres',
"select injection_points_detach('memcontext-server-wait');");
-#Query the same server process again and it should succeed.
+# Query the same server process again and it should succeed.
$topcontext_name = $node->safe_psql('postgres',
"select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
);
@@ -130,7 +133,7 @@ $node->safe_psql('postgres',
"select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
);
-#Client will crash
+# Client will crash
$node->psql(
'postgres',
qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
@@ -143,7 +146,7 @@ like($psql_err,
$node->restart;
$node->poll_query_until('postgres', "SELECT 1;", '1');
-#Querying memory stats should succeed after server start
+# Querying memory stats should succeed after server start
$pid = $node->safe_psql('postgres',
"SELECT pid from pg_stat_activity where backend_type='checkpointer'");
$topcontext_name = $node->safe_psql('postgres',
Attachments:
[text/plain] v52comments.diff.txt (10.7K, ../../[email protected]/2-v52comments.diff.txt)
download | inline diff:
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 120f032f249..aa83b11e131 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -274,10 +274,19 @@
<para>
This function handles requests to display the memory contexts of a
<productname>PostgreSQL</productname> process with the specified
- process ID. The function can be used to send requests to backends as
- well as <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
- If the process does not respond with memory contexts statistics in 5 seconds,
- the function returns NULL.
+ process ID. The function can be used to send requests to
+ <glossterm linkend="glossary-backend">backends</glossterm> as well as
+ <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5
+ seconds, the function returns NULL.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics.
</para>
<para>
The returned record contains extended statistics per each memory
@@ -350,7 +359,8 @@
<listitem>
<para>
<parameter>num_agg_contexts</parameter> - The number of memory
- contexts aggregated in the displayed statistics.
+ contexts aggregated in the displayed statistics. <literal>1</literal>
+ indicates that context statistics are displayed separately.
</para>
</listitem>
</itemizedlist>
@@ -365,16 +375,6 @@
<parameter>summary</parameter> is <literal>false</literal> (the default),
<literal>the num_agg_contexts</literal> value is <literal>1</literal>,
indicating that individual statistics are being displayed.
- </para>
- <para>
- After receiving memory context statistics from the target process, it
- returns the results as one row per context. If all the contexts don't
- fit within the pre-determined size limit, the remaining context
- statistics are aggregated and a cumulative total is displayed. The
- <literal>num_agg_contexts</literal> column indicates the number of
- contexts aggregated in the displayed statistics. A
- <literal>num_agg_contexts</literal> value of <literal>1</literal>
- indicates that context statistics are displayed separately.
</para></entry>
</row>
</tbody>
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index f18b6d47c60..fdf0043211b 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -978,7 +978,11 @@ ProcessGetMemoryContextInterrupt(void)
/*
* The client process should have created the required DSA and DSHash
- * table. Here we just attach to those.
+ * table, so we can connect to these directly. The GetNamedXXX functions
+ * either return DSA/DSHash or error out (which we want to avoid as that
+ * would cause the current transaction to fail) so we don't need to check
+ * for NULL. An assertion is enough to catch in case this API should be
+ * changed at some point.
*/
if (MemoryStatsDsaArea == NULL)
MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
@@ -988,6 +992,8 @@ ProcessGetMemoryContextInterrupt(void)
MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
&memctx_dsh_params, &found);
+ Assert(MemoryStatsDsaArea != NULL && MemoryStatsDsHash != NULL);
+
/*
* The entry lock is held by dshash_find_or_insert to protect writes to
* process specific memory. Two different processes publishing statistics
@@ -1001,9 +1007,8 @@ ProcessGetMemoryContextInterrupt(void)
* if the caller has timed out waiting for us and have issued a request to
* another backend.
*
- * Make sure that the client always deletes the entry after taking
- * required lock or this function may end up writing to unallocated
- * memory.
+ * Make sure that the client always deletes the entry after taking the
+ * required lock or this function may end up writing to unallocated memory.
*/
if (!found || entry->target_server_id != MyProcPid)
{
@@ -1042,13 +1047,11 @@ ProcessGetMemoryContextInterrupt(void)
HASH_ENTER, &found);
Assert(!found);
- /*
- * context id starts with 1
- */
+ /* context id starts with 1 */
contextid_entry->context_id = cxt_id + 1;
/*
- * Copy statistics for each of TopMemoryContexts children. This
+ * Copy statistics for each of TopMemoryContext's children. This
* includes statistics of at most 100 children per node, with each
* child node limited to a depth of 100 in its subtree.
*/
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
index 47bce45bc1d..2ce169ffa69 100644
--- a/src/test/modules/test_memcontext_reporting/Makefile
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -11,8 +11,6 @@ OBJS = \
test_memcontext_reporting.o
PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
-REGRESS = test_memcontext_reporting
-
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
index 73665abf902..152465c1570 100644
--- a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -21,7 +21,6 @@ $node->init;
$node->append_conf(
'postgresql.conf',
qq[
-max_connections = 100
log_statement = none
restart_after_crash = false
]);
@@ -36,7 +35,7 @@ $node->safe_psql('postgres',
my $pid = $node->safe_psql('postgres',
"SELECT pid from pg_stat_activity where backend_type='checkpointer'");
-#Client should have thrown error
+# Client should have thrown error
$node->psql(
'postgres',
qq(select pg_get_process_memory_contexts($pid, true);),
@@ -44,7 +43,8 @@ $node->psql(
like($psql_err,
qr/error triggered for injection point memcontext-client-injection/);
-#Query the same process after detaching the injection point, using some other client and it should succeed.
+# Query the same process after detaching the injection point, using some other
+# client and it should succeed.
$node->safe_psql('postgres',
"select injection_points_detach('memcontext-client-injection');");
my $topcontext_name = $node->safe_psql('postgres',
@@ -57,13 +57,14 @@ $node->safe_psql('postgres',
"select injection_points_attach('memcontext-server-injection', 'error');"
);
-#Server should have thrown error
+# Server should have thrown error
$node->psql(
'postgres',
qq(select pg_get_process_memory_contexts($pid, true);),
stderr => \$psql_err);
-#Query the same process after detaching the injection point, using some other client and it should succeed.
+# Query the same process after detaching the injection point, using some other
+# client and it should succeed.
$node->safe_psql('postgres',
"select injection_points_detach('memcontext-server-injection');");
$topcontext_name = $node->safe_psql('postgres',
@@ -71,8 +72,8 @@ $topcontext_name = $node->safe_psql('postgres',
);
ok($topcontext_name eq 'TopMemoryContext');
-# Test that two concurrent requests to the same process results in a warning for
-# one of those
+# Test that two concurrent requests to the same process results in a warning
+# for one of those
$node->safe_psql('postgres',
"SELECT injection_points_attach('memcontext-client-injection', 'wait');");
@@ -80,8 +81,9 @@ $node->safe_psql('postgres',
"SELECT injection_points_attach('memcontext-server-wait', 'wait');");
my $psql_session1 = $node->background_psql('postgres');
-# We use query_until so that we can execute additional commands without having to
-# wait for this command to complete, since it will pause at injection points.
+# We use query_until so that we can execute additional commands without having
+# to wait for this command to complete, since it will pause at injection
+# points.
$psql_session1->query_until(
qr//,
@@ -95,7 +97,7 @@ $node->psql(
stderr => \$psql_err);
ok($psql_err =~
/WARNING: server process $pid is processing previous request/);
-#Wake the client up.
+# Wake the client up.
$node->safe_psql('postgres',
"SELECT injection_points_wakeup('memcontext-client-injection');");
@@ -104,7 +106,8 @@ $node->safe_psql('postgres',
$node->safe_psql('postgres',
"select injection_points_detach('memcontext-server-wait');");
-# Test the client process exiting with timeout does not break the server process
+# Test the client process exiting with timeout does not break the server
+# process
$node->safe_psql('postgres',
"SELECT injection_points_attach('memcontext-server-wait', 'wait');");
@@ -114,12 +117,12 @@ $node->psql(
qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
stdout => \$psql_out);
is($psql_out, '', 'Query returns null');
-#Wakeup the server process up and detach the injection point.
+# Wakeup the server process up and detach the injection point.
$node->safe_psql('postgres',
"SELECT injection_points_wakeup('memcontext-server-wait');");
$node->safe_psql('postgres',
"select injection_points_detach('memcontext-server-wait');");
-#Query the same server process again and it should succeed.
+# Query the same server process again and it should succeed.
$topcontext_name = $node->safe_psql('postgres',
"select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
);
@@ -130,7 +133,7 @@ $node->safe_psql('postgres',
"select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
);
-#Client will crash
+# Client will crash
$node->psql(
'postgres',
qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
@@ -143,7 +146,7 @@ like($psql_err,
$node->restart;
$node->poll_query_until('postgres', "SELECT 1;", '1');
-#Querying memory stats should succeed after server start
+# Querying memory stats should succeed after server start
$pid = $node->safe_psql('postgres',
"SELECT pid from pg_stat_activity where backend_type='checkpointer'");
$topcontext_name = $node->safe_psql('postgres',
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-02-24 11:57 Rahila Syed <[email protected]>
parent: Daniel Gustafsson <[email protected]>
0 siblings, 1 reply; 65+ messages in thread
From: Rahila Syed @ 2026-02-24 11:57 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: pgsql-hackers; torikoshia <[email protected]>; Robert Haas <[email protected]>; Chao Li <[email protected]>
Hi Daniel,
Thank you for the review. All the changes suggested in the v52comments.diff
are incorporated in the attached patches.
+#Server should have thrown error
> +$node->psql(
> + 'postgres',
> + qq(select pg_get_process_memory_contexts($pid, true);),
> + stderr => \$psql_err);
>
> This test doesn't validate that the server actually errored does it?
> (There is
> no proposed fix in the attached.)
>
>
This has been fixed by adding a check for the error returned by the above
command.
While at it, I also added another crash test to the file, This is similar
to the existing
test for a client backend crash, but in this scenario, it crashes the
server process
instead.
Thank you,
Rahila Syed
Attachments:
[application/octet-stream] v53-0001-Add-function-to-report-memory-context-statistics.patch (60.1K, ../../CAH2L28sH3VoQDHE_kqgrmPMxUq7zdWzUWVB6kg-vu+c0_42MpA@mail.gmail.com/3-v53-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From 5fed1c8eb14eb0bbf9522cecead9dec86fa38a9a Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 159 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/dsm_registry.c | 8 +-
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1023 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 7 +
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
25 files changed, 1315 insertions(+), 22 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 3ac81905d1f..aa83b11e131 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,132 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to
+ <glossterm linkend="glossary-backend">backends</glossterm> as well as
+ <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5
+ seconds, the function returns NULL.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context. The
+ path is limited to 100 children per node, with each node limited
+ to a max depth of 100, to preserve memory during reporting. The
+ printed path will also be limited to 100 nodes counting from the
+ TopMemoryContext.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics. <literal>1</literal>
+ indicates that context statistics are displayed separately.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number of aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +428,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 69699f8830a..b62944cee87 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -378,6 +378,17 @@ BEGIN ATOMIC
END;
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -503,6 +514,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -529,6 +541,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 6fde740465f..3c24119c43a 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -787,6 +787,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..d54193f8890 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -684,6 +684,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index a2c0ff012c5..cb18ce80893 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 82731e452fc..e13a40f1427 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -870,6 +870,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..1d6d8f22d20 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 742137edad6..d5a3cb13a4c 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -875,6 +875,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/dsm_registry.c b/src/backend/storage/ipc/dsm_registry.c
index 068c1577b12..4810babac1b 100644
--- a/src/backend/storage/ipc/dsm_registry.c
+++ b/src/backend/storage/ipc/dsm_registry.c
@@ -270,7 +270,9 @@ GetNamedDSMSegment(const char *name, size_t size,
* This routine returns a pointer to the DSA. A new LWLock tranche ID will be
* generated if needed. Note that the lock tranche will be registered with the
* provided name. Also note that this should be called at most once for a
- * given DSA in each backend.
+ * given DSA in each backend. It's best to minimize the number of errors thrown
+ * in this function since it's called from a CFI function. An error here could
+ * lead to error in any transaction that calls the CFI function.
*/
dsa_area *
GetNamedDSA(const char *name, bool *found)
@@ -351,7 +353,9 @@ GetNamedDSA(const char *name, bool *found)
* params is ignored; a new LWLock tranche ID will be generated if needed.
* Note that the lock tranche will be registered with the provided name. Also
* note that this should be called at most once for a given table in each
- * backend.
+ * backend. It's best to minimize the number of errors thrown in this function
+ * since it's called from a CFI function. An error here could lead to error in
+ * any transaction that calls the CFI function.
*/
dshash_table *
GetNamedDSHash(const char *name, const dshash_parameters *params, bool *found)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 1f7e933d500..b0176613987 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -141,6 +142,7 @@ CalculateShmemSize(void)
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
size = add_size(size, LogicalDecodingCtlShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
AioShmemInit();
WaitLSNShmemInit();
LogicalDecodingCtlShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index 7505c9d3a37..839cbdbdc89 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -695,6 +695,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index d01a09dd0c4..9e0d289a604 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3573,6 +3573,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 4aa864fe3c3..3e1dc4225b2 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -163,6 +163,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -366,6 +367,7 @@ SerialControl "Waiting to read or update shared <filename>pg_serial</filename> s
AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
WaitLSN "Waiting to read or update shared Wait-for-LSN state."
LogicalDecodingControl "Waiting to read or update logical decoding status information."
+MemContextReportingClientKeys "Waiting for another process to complete reading or writing the memory reporting keys."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c7f7b8bc2dd..fdf0043211b 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,25 +15,128 @@
#include "postgres.h"
+#include "access/xact.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "storage/procsignal.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+#define MAX_PATH_DISPLAY_LENGTH 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[MAX_PATH_DISPLAY_LENGTH];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ ProcNumber client_key;
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ offsetof(MemoryStatsDSHashEntry, memcxt_cv),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_memcmp,
+ dshash_memhash,
+ dshash_memcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(int ProcNumber);
+static void memstats_client_key_reset(int ProcNumber);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -63,6 +166,67 @@ int_list_to_array(const List *list)
return PointerGetDatum(result_array);
}
+/*
+ * context_type_to_string
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+static const char *
+context_type_to_string(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ {
+ elog(NOTICE, "hash table corrupted, can't construct path value");
+ return NIL;
+ }
+ path = lcons_int(cur_entry->context_id, path);
+ }
+ return path;
+}
+
/*
* PutMemoryContextsStatsTupleStore
* Add details for the given MemoryContext to 'tupstore'.
@@ -144,24 +308,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = context_type_to_string(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -306,3 +453,839 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber client_procno = MyProcNumber;
+ ProcNumber server_procno;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ server_procno = GetNumberFromPGProc(proc);
+
+ /*
+ * First, check if the server process slot is occupied; if so, exit early.
+ * An occupied slot indicates another client backend is already requesting
+ * statistics from this server process. We should not set the slot yet,
+ * even if it appears empty, until we are prepared to signal the server
+ * process. This is because if the following DSA APIs throw an error, the
+ * server slot may not be reset, leading to it being incorrectly marked as
+ * occupied and blocking further memory context statistics queries to the
+ * same process.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[server_procno] != -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &client_procno, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[server_procno] == -1)
+ client_keys[server_procno] = MyProcNumber;
+ else
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ memstats_dsa_cleanup(client_procno);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, server_procno) < 0)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for at most MEMORY_STATS_MAX_TIMEOUT seconds, if we have been
+ * woken up spuriously we subtract the time already slept. If no
+ * statistics are available within the allowed time then return NULL.
+ * The timer is defined in milliseconds since that's what the
+ * condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000) - elapsed_time,
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics for PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &client_procno, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(context_type_to_string(memcxt_info[i].type));
+
+ if (memcxt_info[i].levels != 0)
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+ else
+ nulls[3] = true;
+
+ if (memcxt_info[i].path[0] != 0)
+ {
+ int path_length;
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(client_procno);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+/*
+ * This function is executed before returning from the
+ * pg_get_process_memory_contexts() function.
+ * It releases the DSA memory allocated for storing the memory statistics
+ * retrieved by that function. The caller must ensure that the LWLock for
+ * MemoryStatsDsHash is not already held.
+*/
+static void
+memstats_dsa_cleanup(int procNumber)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, &procNumber, true);
+
+ Assert(MemoryStatsDsaArea != NULL);
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize(), &found);
+
+ if (!found)
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+
+ return mul_size(TotalProcs, sizeof(int));
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Avoid performing any shared memory operations in aborted transaction,
+ * the caller will timeout and return NULL.
+ */
+ if (IsAbortedTransactionBlockState())
+ return;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ return;
+ }
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table, so we can connect to these directly. The GetNamedXXX functions
+ * either return DSA/DSHash or error out (which we want to avoid as that
+ * would cause the current transaction to fail) so we don't need to check
+ * for NULL. An assertion is enough to catch in case this API should be
+ * changed at some point.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ Assert(MemoryStatsDsaArea != NULL && MemoryStatsDsHash != NULL);
+
+ /*
+ * The entry lock is held by dshash_find_or_insert to protect writes to
+ * process specific memory. Two different processes publishing statistics
+ * do not block each other.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &clientProcNumber, &found);
+ INJECTION_POINT("memcontext-server-injection", NULL);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking the
+ * required lock or this function may end up writing to unallocated memory.
+ */
+ if (!found || entry->target_server_id != MyProcPid)
+ {
+ entry->stats_written = false;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /* context id starts with 1 */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContext's children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].levels = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strncmp(context->name, "dynahash", 8) == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &idx, &found);
+
+ if (found)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ }
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..e246fc7ce1f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b59e08605cc..ba5d3bf0d7a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -669,6 +669,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 073bdb35d2a..2a826cbef17 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1011,6 +1011,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ Assert(totals != NULL);
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index dac40992cbc..3667bc2bf41 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8641,6 +8641,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified process',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'u',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9..38e1e55a9a6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index e94ebce95b9..6dd80689bbb 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -87,6 +87,7 @@ PG_LWLOCK(52, SerialControl)
PG_LWLOCK(53, AioWorkerSubmissionQueue)
PG_LWLOCK(54, WaitLSN)
PG_LWLOCK(55, LogicalDecodingControl)
+PG_LWLOCK(56, MemContextReportingClientKeys)
/*
* There also exist several built-in LWLock tranches. As with the predefined
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 348fba53a93..00ccfcb22a4 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
PROCSIG_RECOVERY_CONFLICT, /* backend is blocking recovery, check
* PGPROC->pendingRecoveryConflicts for the
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 11ab1717a16..40099036308 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -319,4 +319,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3dd63fd88ed..c84e5adce16 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 004f9a70e00..d34d2b6211c 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 241945734ec..f90ff78a78b 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1709,6 +1709,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
[application/octet-stream] v53-0002-Test-module-to-test-memory-context-reporting-wit.patch (10.1K, ../../CAH2L28sH3VoQDHE_kqgrmPMxUq7zdWzUWVB6kg-vu+c0_42MpA@mail.gmail.com/4-v53-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From d7902195131dad7ee1d08be6213f31e65f754ed3 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
.../test_memcontext_reporting/Makefile | 23 +++
.../t/001_memcontext_inj.pl | 185 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 4 +
5 files changed, 225 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 44c7163c1cd..8ce9598495c 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -34,6 +34,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..2ce169ffa69
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+TAP_TESTS = 1
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..947059ceece
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,185 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+# Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+# Query the same process after detaching the injection point, using some other
+# client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+# Server process throws an error, which causes client to time out
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+like($psql_err,
+ qr/NOTICE: request for memory context statistics for PID $pid timed out/);
+# Query the same process after detaching the injection point, using some other
+# client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning
+# for one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+
+# We use query_until so that we can execute additional commands without having
+# to wait for this command to complete, since it will pause at injection
+# points.
+
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+# Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server
+# process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+is($psql_out, '', 'Query returns null');
+# Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+# Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test if the monitoring works fine when server restarts after client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+# Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+# Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test if the monitoring works fine when server restarts after server process crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+# Server process will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+# Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..48b501682d5
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1,4 @@
+comment = 'Test code for memcontext reporting'
+default_version = '1.0'
+module_pathname = '$libdir/test_memcontext_reporting'
+relocatable = true
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
* Re: Enhancing Memory Context Statistics Reporting
@ 2026-03-05 22:31 Rahila Syed <[email protected]>
parent: Rahila Syed <[email protected]>
0 siblings, 0 replies; 65+ messages in thread
From: Rahila Syed @ 2026-03-05 22:31 UTC (permalink / raw)
To: Daniel Gustafsson <[email protected]>; +Cc: pgsql-hackers; torikoshia <[email protected]>; Robert Haas <[email protected]>; Chao Li <[email protected]>
Hi All,
The email below provides a summary of the changes implemented since the
previous
update on December 18th.
1. I tested and verified that when ProcessGetMemoryContextInterrupt()
returns with error
from processes that don't start a transaction, it doesn't affect system
integrity.
In auxiliary processes like ioworker, archiver, and wal_receiver, the ERROR
causes
the process to exit, which ensures proper resource and lock release. In
other cases,
cleanup and release are managed by error recovery logic, allowing processes
to continue
running smoothly.
2. I verified that the only resources acquired in
ProcessGetMemoryContextInterrupt
are DSMs, which remain attached until the process exits.
During proc_exit, the release/detach of DSM runs via dsm_backend_shutdown().
3.The error reporting level in the interrupt handler has been adjusted from
ERROR
to NOTICE to prevent unnecessary errors from being raised in interrupted
transactions
due to minor issues.
4. Aborted transaction check has been added to allow returning without
processing
the interrupt if it is called from an aborted transaction.
5. DSHash key type has been changed from string to integer, now using
procNumber
directly. This eliminates string formatting overhead.
6. Timeout messaging has been changed from silently returning empty row to
issuing
a NOTICE and returning an empty row.
7. Documentation has been to GetNamedDSA() and GetNamedDSHash(), warning
that
errors thrown from these functions can propagate to any transaction calling
the CFI function
8. Subtracted elapsed time ((MEMORY_STATS_MAX_TIMEOUT * 1000) -
elapsed_time) from
timeout passed to ConditionVariableTimedSleep so the total wait doesn't
exceed the intended
timeout in case of spurious wakeups.
9. Fixed a race condition bug by ensuring ConditionVariableSignal is called
before
dshash_release_lock
Additionally, I included couple of assertions, renamed variables and
functions to enhance
clarity and consistency with the existing style, and made improvements to
the documentation
and tests.
Please find attached updated and rebased patches.
Thank you,
Rahila Syed
On Tue, Feb 24, 2026 at 5:27 PM Rahila Syed <[email protected]> wrote:
> Hi Daniel,
>
> Thank you for the review. All the changes suggested in the v52comments.diff
> are incorporated in the attached patches.
>
> +#Server should have thrown error
>> +$node->psql(
>> + 'postgres',
>> + qq(select pg_get_process_memory_contexts($pid, true);),
>> + stderr => \$psql_err);
>>
>> This test doesn't validate that the server actually errored does it?
>> (There is
>> no proposed fix in the attached.)
>>
>>
> This has been fixed by adding a check for the error returned by the above
> command.
> While at it, I also added another crash test to the file, This is similar
> to the existing
> test for a client backend crash, but in this scenario, it crashes the
> server process
> instead.
>
> Thank you,
> Rahila Syed
>
Attachments:
[application/octet-stream] v54-0002-Test-module-to-test-memory-context-reporting-wit.patch (10.4K, ../../CAH2L28tU+aC1+OR5S1ATxHPeyuGp5JAfBVcs+7r-h_N2JRaxUw@mail.gmail.com/3-v54-0002-Test-module-to-test-memory-context-reporting-wit.patch)
download | inline diff:
From d6d4022de834e714820e85f928d2e734197d994a Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Fri, 28 Nov 2025 14:46:38 +0530
Subject: [PATCH 2/2] Test module to test memory context reporting with
injection points
---
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
.../test_memcontext_reporting/Makefile | 23 +++
.../t/001_memcontext_inj.pl | 185 ++++++++++++++++++
.../test_memcontext_reporting.c | 12 ++
.../test_memcontext_reporting.control | 1 +
6 files changed, 223 insertions(+)
create mode 100644 src/test/modules/test_memcontext_reporting/Makefile
create mode 100644 src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
create mode 100644 src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile
index 4ac5c84db43..4cb276e40f0 100644
--- a/src/test/modules/Makefile
+++ b/src/test/modules/Makefile
@@ -35,6 +35,7 @@ SUBDIRS = \
test_json_parser \
test_lfind \
test_lwlock_tranches \
+ test_memcontext_reporting \
test_misc \
test_oat_hooks \
test_parser \
diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build
index e2b3eef4136..e785ea7e1f9 100644
--- a/src/test/modules/meson.build
+++ b/src/test/modules/meson.build
@@ -36,6 +36,7 @@ subdir('test_integerset')
subdir('test_json_parser')
subdir('test_lfind')
subdir('test_lwlock_tranches')
+subdir('test_memcontext_reporting')
subdir('test_misc')
subdir('test_oat_hooks')
subdir('test_parser')
diff --git a/src/test/modules/test_memcontext_reporting/Makefile b/src/test/modules/test_memcontext_reporting/Makefile
new file mode 100644
index 00000000000..2ce169ffa69
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/Makefile
@@ -0,0 +1,23 @@
+# src/test/modules/test_memcontext_reporting/Makefile
+
+TAP_TESTS = 1
+
+EXTRA_INSTALL = src/test/modules/injection_points
+
+export enable_injection_points
+MODULE_big = test_memcontext_reporting
+OBJS = \
+ $(WIN32RES) \
+ test_memcontext_reporting.o
+PGFILEDESC = "test_memcontext_reporting - test code for memory context reporting"
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = src/test/modules/test_memcontext_reporting
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
diff --git a/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
new file mode 100644
index 00000000000..e2e4d197e2b
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/t/001_memcontext_inj.pl
@@ -0,0 +1,185 @@
+# Copyright (c) 2026, PostgreSQL Global Development Group
+
+# Test suite for testing memory context statistics reporting
+
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+my $psql_err;
+my $psql_out;
+# Create and start a cluster with one node
+my $node = PostgreSQL::Test::Cluster->new('main');
+$node->init;
+$node->append_conf(
+ 'postgresql.conf',
+ qq[
+log_statement = none
+restart_after_crash = false
+]);
+$node->start;
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points;');
+
+# Attaching to a client process's injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'error');"
+);
+
+my $pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+
+# Client should have thrown error
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/error triggered for injection point memcontext-client-injection/);
+
+# Query the same process after detaching the injection point, using some other
+# client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+my $topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Attaching to a target process injection point that throws an error
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'error');"
+);
+
+# Server process throws an error, which causes client to time out
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+
+like($psql_err,
+ qr/NOTICE: request for memory context statistics to PID $pid timed out/);
+# Query the same process after detaching the injection point, using some other
+# client and it should succeed.
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-injection');");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test that two concurrent requests to the same process results in a warning
+# for one of those
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-client-injection', 'wait');");
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+my $psql_session1 = $node->background_psql('postgres');
+
+# We use query_until so that we can execute additional commands without having
+# to wait for this command to complete, since it will pause at injection
+# points.
+
+$psql_session1->query_until(
+ qr//,
+ qq(
+ SELECT pg_get_process_memory_contexts($pid, true);
+));
+$node->wait_for_event('client backend', 'memcontext-client-injection');
+$node->psql(
+ 'postgres',
+ qq(select pg_get_process_memory_contexts($pid, true);),
+ stderr => \$psql_err);
+ok($psql_err =~
+ /WARNING: server process $pid is processing previous request/);
+# Wake the client up.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-client-injection');");
+
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-client-injection');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+
+# Test the client process exiting with timeout does not break the server
+# process
+
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('memcontext-server-wait', 'wait');");
+# Following client query times out, returning NULL as output
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stdout => \$psql_out);
+is($psql_out, '', 'Query returns null');
+# Wakeup the server process up and detach the injection point.
+$node->safe_psql('postgres',
+ "SELECT injection_points_wakeup('memcontext-server-wait');");
+$node->safe_psql('postgres',
+ "select injection_points_detach('memcontext-server-wait');");
+# Query the same server process again and it should succeed.
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test if the monitoring works fine when server restarts after client backend crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-client-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+# Client will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+# Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+# Test if the monitoring works fine when server restarts after server process crashes.
+$node->safe_psql('postgres',
+ "select injection_points_attach('memcontext-server-injection', 'test_memcontext_reporting', 'crash', NULL);"
+);
+
+# Server process will crash
+$node->psql(
+ 'postgres',
+ qq(select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';),
+ stderr => \$psql_err);
+like($psql_err,
+ qr/WARNING: terminating connection because of crash of another server process|server closed the connection unexpectedly|connection to server was lost|could not send data to server/
+);
+
+# Wait till server restarts
+$node->restart;
+$node->poll_query_until('postgres', "SELECT 1;", '1');
+
+# Querying memory stats should succeed after server start
+$pid = $node->safe_psql('postgres',
+ "SELECT pid from pg_stat_activity where backend_type='checkpointer'");
+$topcontext_name = $node->safe_psql('postgres',
+ "select name from pg_get_process_memory_contexts($pid, true) where path = '{1}';"
+);
+ok($topcontext_name eq 'TopMemoryContext');
+
+done_testing();
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
new file mode 100644
index 00000000000..d641f3616dc
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.c
@@ -0,0 +1,12 @@
+#include "postgres.h"
+#include "funcapi.h"
+
+PG_MODULE_MAGIC;
+
+extern PGDLLEXPORT void crash(const char *name, const void *private_data, void *arg);
+
+void
+crash(const char *name, const void *private_data, void *arg)
+{
+ abort();
+}
diff --git a/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
new file mode 100644
index 00000000000..10c3786d193
--- /dev/null
+++ b/src/test/modules/test_memcontext_reporting/test_memcontext_reporting.control
@@ -0,0 +1 @@
+comment = 'Test code for memcontext reporting'
--
2.34.1
[application/octet-stream] v54-0001-Add-function-to-report-memory-context-statistics.patch (60.3K, ../../CAH2L28tU+aC1+OR5S1ATxHPeyuGp5JAfBVcs+7r-h_N2JRaxUw@mail.gmail.com/4-v54-0001-Add-function-to-report-memory-context-statistics.patch)
download | inline diff:
From 836a1b255418a23be3378a826a06bb1c8248ec96 Mon Sep 17 00:00:00 2001
From: Rahila Syed <[email protected]>
Date: Thu, 27 Nov 2025 14:39:43 +0530
Subject: [PATCH 1/2] Add function to report memory context statistics
This adds a function for retrieving memory context statistics
and information from backends as well as auxiliary processes.
The intended usecase is cluster debugging when under memory
pressure or unanticipated memory usage characteristics.
When calling the function it sends a signal to the specified
process to submit statistics regarding its memory contexts
into dynamic shared memory. Each memory context is returned
in detail, followed by a cumulative total in case the number
of contexts exceed the max allocated amount of shared memory.
Each process is limited to use at most 1Mb memory for this.
A summary can also be explicitly requested by the user, this
will return the TopMemoryContext and a cumulative total of
all lower contexts.
In order to not block on busy processes, we have hardcoded
the number of seconds during which to retry before timing out.
In the case where no statistics are published within the set
timeout, NULL is returned
---
doc/src/sgml/func/func-admin.sgml | 160 +++
src/backend/catalog/system_functions.sql | 14 +
src/backend/postmaster/autovacuum.c | 4 +
src/backend/postmaster/checkpointer.c | 4 +
src/backend/postmaster/interrupt.c | 4 +
src/backend/postmaster/pgarch.c | 4 +
src/backend/postmaster/startup.c | 4 +
src/backend/postmaster/walsummarizer.c | 4 +
src/backend/storage/ipc/dsm_registry.c | 8 +-
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/ipc/procsignal.c | 3 +
src/backend/tcop/postgres.c | 3 +
.../utils/activity/wait_event_names.txt | 2 +
src/backend/utils/adt/mcxtfuncs.c | 1025 ++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 7 +
src/backend/utils/mmgr/mcxt.c | 31 +
src/include/catalog/pg_proc.dat | 10 +
src/include/miscadmin.h | 1 +
src/include/storage/lwlocklist.h | 1 +
src/include/storage/procsignal.h | 1 +
src/include/utils/memutils.h | 7 +
src/test/regress/expected/sysviews.out | 19 +
src/test/regress/sql/sysviews.sql | 18 +
src/tools/pgindent/typedefs.list | 2 +
25 files changed, 1318 insertions(+), 22 deletions(-)
diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml
index 17319f9f969..06bdfc8ad12 100644
--- a/doc/src/sgml/func/func-admin.sgml
+++ b/doc/src/sgml/func/func-admin.sgml
@@ -251,6 +251,133 @@
<literal>false</literal> is returned.
</para></entry>
</row>
+
+ <row>
+ <entry role="func_table_entry"><para role="func_signature">
+ <indexterm>
+ <primary>pg_get_process_memory_contexts</primary>
+ </indexterm>
+ <function>pg_get_process_memory_contexts</function> ( <parameter>pid</parameter> <type>integer</type> <optional>,<parameter>summary</parameter> <type>boolean</type> <literal>DEFAULT</literal> <literal>false</literal></optional> )
+ <returnvalue>setof record</returnvalue>
+ ( <parameter>name</parameter> <type>text</type>,
+ <parameter>ident</parameter> <type>text</type>,
+ <parameter>type</parameter> <type>text</type>,
+ <parameter>level</parameter> <type>integer</type>,
+ <parameter>path</parameter> <type>integer[]</type>,
+ <parameter>total_bytes</parameter> <type>bigint</type>,
+ <parameter>total_nblocks</parameter> <type>bigint</type>,
+ <parameter>free_bytes</parameter> <type>bigint</type>,
+ <parameter>free_chunks</parameter> <type>bigint</type>,
+ <parameter>used_bytes</parameter> <type>bigint</type>,
+ <parameter>num_agg_contexts</parameter> <type>integer</type> )
+ </para>
+ <para>
+ This function handles requests to display the memory contexts of a
+ <productname>PostgreSQL</productname> process with the specified
+ process ID. The function can be used to send requests to
+ <glossterm linkend="glossary-backend">backends</glossterm> as well as
+ <glossterm linkend="glossary-auxiliary-proc">auxiliary processes</glossterm>.
+ If the process does not respond with memory contexts statistics in 5
+ seconds, the function emits a <literal>NOTICE</literal> with the message
+ <literal>request for memory context statistics to PID %d timed out</literal>,
+ and then returns NULL.
+ </para>
+ <para>
+ After receiving memory context statistics from the target process, it
+ returns the results as one row per context. If all the contexts don't
+ fit within the pre-determined size limit, the remaining context
+ statistics are aggregated and a cumulative total is displayed. The
+ <literal>num_agg_contexts</literal> column indicates the number of
+ contexts aggregated in the displayed statistics.
+ </para>
+ <para>
+ The returned record contains extended statistics per each memory
+ context:
+ <itemizedlist spacing="compact">
+ <listitem>
+ <para>
+ <parameter>name</parameter> - The name of the memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>ident</parameter> - Memory context ID (if any).
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>type</parameter> - The type of memory context, possible
+ values are: AllocSet, Generation, Slab and Bump.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>level</parameter> - The level in the tree of the current
+ memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>path</parameter> - Memory contexts are organized in a
+ tree model with TopMemoryContext as the root, and all other memory
+ contexts as nodes in the tree. The <parameter>path</parameter>
+ displays the path from the root to the current memory context
+ The path that is printed will be limited to 100 nodes counting from
+ the TopMemoryContext, to preserve memory during reporting.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_bytes</parameter> - The total number of bytes
+ allocated to this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>total_nblocks</parameter> - The total number of blocks
+ used for the allocated memory.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_bytes</parameter> - The amount of free memory in
+ this memory context.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>free_chunks</parameter> - The number of chunks that
+ <parameter>free_bytes</parameter> corresponds to.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>used_bytes</parameter> - The total number of bytes
+ currently occupied.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <parameter>num_agg_contexts</parameter> - The number of memory
+ contexts aggregated in the displayed statistics. Statistics are aggregated
+ when the internal reporting buffer (about 1 MB) is filled.<literal>1</literal>
+ indicates that context statistics are displayed separately.
+ </para>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ When <parameter>summary</parameter> is <literal>true</literal>, statistics
+ for memory contexts at levels 1 and 2 are displayed, with level 1
+ representing the root node (i.e., <literal>TopMemoryContext</literal>).
+ Statistics for contexts on level 2 and below are aggregates of all
+ child contexts' statistics, where <literal>num_agg_contexts</literal>
+ indicate the number of aggregated child contexts. When
+ <parameter>summary</parameter> is <literal>false</literal> (the default),
+ <literal>the num_agg_contexts</literal> value is <literal>1</literal>,
+ indicating that individual statistics are being displayed.
+ </para></entry>
+ </row>
</tbody>
</tgroup>
</table>
@@ -302,6 +429,39 @@ LOG: Grand total: 1651920 bytes in 201 blocks; 622360 free (88 chunks); 1029560
because it may generate a large number of log messages.
</para>
+ <para>
+ <function>pg_get_process_memory_contexts</function> can be used to request
+ memory contexts statistics of any <productname>PostgreSQL</productname>
+ process. For example:
+<programlisting>
+postgres=# SELECT * FROM pg_get_process_memory_contexts(
+ (SELECT pid FROM pg_stat_activity
+ WHERE backend_type = 'checkpointer'),
+ false) LIMIT 1;
+-[ RECORD 1 ]----+------------------------------
+name | TopMemoryContext
+ident |
+type | AllocSet
+level | 1
+path | {1}
+total_bytes | 90304
+total_nblocks | 3
+free_bytes | 2880
+free_chunks | 1
+used_bytes | 87424
+num_agg_contexts | 1
+</programlisting>
+ <note>
+ <para>
+ While <function>pg_get_process_memory_contexts</function> can be used to
+ query memory contexts of the local backend,
+ <structname>pg_backend_memory_contexts</structname>
+ (see <xref linkend="view-pg-backend-memory-contexts"/> for more details)
+ will be less resource intensive when only the local backend is of interest.
+ </para>
+ </note>
+ </para>
+
</sect2>
<sect2 id="functions-admin-backup">
diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql
index 69699f8830a..b62944cee87 100644
--- a/src/backend/catalog/system_functions.sql
+++ b/src/backend/catalog/system_functions.sql
@@ -378,6 +378,17 @@ BEGIN ATOMIC
END;
+CREATE OR REPLACE FUNCTION
+ pg_get_process_memory_contexts(IN pid integer, IN summary boolean DEFAULT false,
+ OUT name text, OUT ident text, OUT type text, OUT level integer,
+ OUT path integer[], OUT total_bytes bigint, OUT total_nblocks bigint,
+ OUT free_bytes bigint, OUT free_chunks bigint, OUT used_bytes bigint,
+ OUT num_agg_contexts integer)
+RETURNS SETOF RECORD
+LANGUAGE INTERNAL
+STRICT VOLATILE PARALLEL UNSAFE
+AS 'pg_get_process_memory_contexts';
+
--
-- The default permissions for functions mean that anyone can execute them.
-- A number of functions shouldn't be executable by just anyone, but rather
@@ -503,6 +514,7 @@ REVOKE EXECUTE ON FUNCTION pg_ls_logicalmapdir() FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_ls_replslotdir(text) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) FROM PUBLIC;
--
-- We also set up some things as accessible to standard roles.
--
@@ -529,6 +541,8 @@ GRANT EXECUTE ON FUNCTION pg_current_logfile() TO pg_monitor;
GRANT EXECUTE ON FUNCTION pg_current_logfile(text) TO pg_monitor;
+GRANT EXECUTE ON FUNCTION pg_get_process_memory_contexts(integer, boolean) TO pg_read_all_stats;
+
GRANT pg_read_all_settings TO pg_monitor;
GRANT pg_read_all_stats TO pg_monitor;
diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c
index 6fde740465f..3c24119c43a 100644
--- a/src/backend/postmaster/autovacuum.c
+++ b/src/backend/postmaster/autovacuum.c
@@ -787,6 +787,10 @@ ProcessAutoVacLauncherInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
/* Process sinval catchup interrupts that happened while sleeping */
ProcessCatchupInterrupt();
}
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index e03c19123bc..d54193f8890 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -684,6 +684,10 @@ ProcessCheckpointerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c
index a2c0ff012c5..cb18ce80893 100644
--- a/src/backend/postmaster/interrupt.c
+++ b/src/backend/postmaster/interrupt.c
@@ -48,6 +48,10 @@ ProcessMainLoopInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c
index 82731e452fc..e13a40f1427 100644
--- a/src/backend/postmaster/pgarch.c
+++ b/src/backend/postmaster/pgarch.c
@@ -870,6 +870,10 @@ ProcessPgArchInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ConfigReloadPending)
{
char *archiveLib = pstrdup(XLogArchiveLibrary);
diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c
index cdbe53dd262..1d6d8f22d20 100644
--- a/src/backend/postmaster/startup.c
+++ b/src/backend/postmaster/startup.c
@@ -192,6 +192,10 @@ ProcessStartupProcInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
diff --git a/src/backend/postmaster/walsummarizer.c b/src/backend/postmaster/walsummarizer.c
index 742137edad6..d5a3cb13a4c 100644
--- a/src/backend/postmaster/walsummarizer.c
+++ b/src/backend/postmaster/walsummarizer.c
@@ -875,6 +875,10 @@ ProcessWalSummarizerInterrupts(void)
/* Perform logging of memory contexts of this process */
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+
+ /* Publish memory contexts of this process */
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
}
/*
diff --git a/src/backend/storage/ipc/dsm_registry.c b/src/backend/storage/ipc/dsm_registry.c
index 068c1577b12..4810babac1b 100644
--- a/src/backend/storage/ipc/dsm_registry.c
+++ b/src/backend/storage/ipc/dsm_registry.c
@@ -270,7 +270,9 @@ GetNamedDSMSegment(const char *name, size_t size,
* This routine returns a pointer to the DSA. A new LWLock tranche ID will be
* generated if needed. Note that the lock tranche will be registered with the
* provided name. Also note that this should be called at most once for a
- * given DSA in each backend.
+ * given DSA in each backend. It's best to minimize the number of errors thrown
+ * in this function since it's called from a CFI function. An error here could
+ * lead to error in any transaction that calls the CFI function.
*/
dsa_area *
GetNamedDSA(const char *name, bool *found)
@@ -351,7 +353,9 @@ GetNamedDSA(const char *name, bool *found)
* params is ignored; a new LWLock tranche ID will be generated if needed.
* Note that the lock tranche will be registered with the provided name. Also
* note that this should be called at most once for a given table in each
- * backend.
+ * backend. It's best to minimize the number of errors thrown in this function
+ * since it's called from a CFI function. An error here could lead to error in
+ * any transaction that calls the CFI function.
*/
dshash_table *
GetNamedDSHash(const char *name, const dshash_parameters *params, bool *found)
diff --git a/src/backend/storage/ipc/ipci.c b/src/backend/storage/ipc/ipci.c
index 1f7e933d500..b0176613987 100644
--- a/src/backend/storage/ipc/ipci.c
+++ b/src/backend/storage/ipc/ipci.c
@@ -52,6 +52,7 @@
#include "storage/sinvaladt.h"
#include "utils/guc.h"
#include "utils/injection_point.h"
+#include "utils/memutils.h"
/* GUCs */
int shared_memory_type = DEFAULT_SHARED_MEMORY_TYPE;
@@ -141,6 +142,7 @@ CalculateShmemSize(void)
size = add_size(size, AioShmemSize());
size = add_size(size, WaitLSNShmemSize());
size = add_size(size, LogicalDecodingCtlShmemSize());
+ size = add_size(size, MemoryContextKeysShmemSize());
/* include additional requested shmem from preload libraries */
size = add_size(size, total_addin_request);
@@ -328,6 +330,7 @@ CreateOrAttachShmemStructs(void)
AioShmemInit();
WaitLSNShmemInit();
LogicalDecodingCtlShmemInit();
+ MemoryContextKeysShmemInit();
}
/*
diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c
index d47d180a32f..227d6333632 100644
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -696,6 +696,9 @@ procsignal_sigusr1_handler(SIGNAL_ARGS)
if (CheckProcSignal(PROCSIG_LOG_MEMORY_CONTEXT))
HandleLogMemoryContextInterrupt();
+ if (CheckProcSignal(PROCSIG_GET_MEMORY_CONTEXT))
+ HandleGetMemoryContextInterrupt();
+
if (CheckProcSignal(PROCSIG_PARALLEL_APPLY_MESSAGE))
HandleParallelApplyMessageInterrupt();
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index d01a09dd0c4..9e0d289a604 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -3573,6 +3573,9 @@ ProcessInterrupts(void)
if (LogMemoryContextPending)
ProcessLogMemoryContextInterrupt();
+ if (PublishMemoryContextPending)
+ ProcessGetMemoryContextInterrupt();
+
if (ParallelApplyMessagePending)
ProcessParallelApplyMessages();
}
diff --git a/src/backend/utils/activity/wait_event_names.txt b/src/backend/utils/activity/wait_event_names.txt
index 4aa864fe3c3..3e1dc4225b2 100644
--- a/src/backend/utils/activity/wait_event_names.txt
+++ b/src/backend/utils/activity/wait_event_names.txt
@@ -163,6 +163,7 @@ WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at transaction end."
+MEM_CXT_PUBLISH "Waiting for a process to publish memory information."
ABI_compatibility:
@@ -366,6 +367,7 @@ SerialControl "Waiting to read or update shared <filename>pg_serial</filename> s
AioWorkerSubmissionQueue "Waiting to access AIO worker submission queue."
WaitLSN "Waiting to read or update shared Wait-for-LSN state."
LogicalDecodingControl "Waiting to read or update logical decoding status information."
+MemContextReportingClientKeys "Waiting for another process to complete reading or writing the memory reporting keys."
#
# END OF PREDEFINED LWLOCKS (DO NOT CHANGE THIS LINE)
diff --git a/src/backend/utils/adt/mcxtfuncs.c b/src/backend/utils/adt/mcxtfuncs.c
index c7f7b8bc2dd..f0bc634965f 100644
--- a/src/backend/utils/adt/mcxtfuncs.c
+++ b/src/backend/utils/adt/mcxtfuncs.c
@@ -15,25 +15,128 @@
#include "postgres.h"
+#include "access/xact.h"
#include "funcapi.h"
#include "mb/pg_wchar.h"
+#include "storage/dsm_registry.h"
#include "storage/proc.h"
#include "storage/procarray.h"
#include "storage/procsignal.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/hsearch.h"
+#include "utils/injection_point.h"
+#include "utils/memutils.h"
+#include "utils/wait_event_types.h"
+
+/*
+ * Memory Context reporting size limits.
+ */
+
+/* Max length of context name and ident, to keep it consistent
+ * with ProcessLogMemoryContext()
+ */
+#define MEMORY_CONTEXT_IDENT_SHMEM_SIZE 100
+#define MEMORY_CONTEXT_NAME_SHMEM_SIZE 100
+#define MAX_PATH_DISPLAY_LENGTH 100
+
+/* Maximum size (in bytes) of DSA area per process */
+#define MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND ((size_t) (1 * 1024 * 1024))
+
+/*
+ * Maximum number of memory context statistics is calculated by dividing
+ * max memory allocated per backend with maximum size per context statistics.
+ * The identifier and name are statically allocated arrays of size 100 bytes.
+ * The path depth is limited to 100 like for memory context logging.
+ */
+#define MAX_MEMORY_CONTEXT_STATS_NUM MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND / (sizeof(MemoryStatsEntry))
+
+/*
+ * Size of dshash key. The key is a uint32 rendered as a string, 10 chars
+ * plus space for a NULL terminator can hold all the values.
+ */
+#define CLIENT_KEY_SIZE (10 + 1)
+
+/* Dynamic shared memory state for reporting statistics per context */
+typedef struct MemoryStatsEntry
+{
+ char name[MEMORY_CONTEXT_NAME_SHMEM_SIZE];
+ char ident[MEMORY_CONTEXT_IDENT_SHMEM_SIZE];
+ int path[MAX_PATH_DISPLAY_LENGTH];
+ NodeTag type;
+ int path_length;
+ int levels;
+ int64 totalspace;
+ int64 nblocks;
+ int64 freespace;
+ int64 freechunks;
+ int num_agg_stats;
+} MemoryStatsEntry;
+
+/*
+ * Per backend dynamic shared hash entry for memory context statistics
+ * reporting.
+ */
+typedef struct MemoryStatsDSHashEntry
+{
+ ProcNumber client_key;
+ ConditionVariable memcxt_cv;
+ bool stats_written;
+ int target_server_id;
+ int total_stats;
+ bool summary;
+ dsa_pointer memstats_dsa_pointer;
+} MemoryStatsDSHashEntry;
+
+static const dshash_parameters memctx_dsh_params = {
+ sizeof(ProcNumber),
+ sizeof(MemoryStatsDSHashEntry),
+ dshash_memcmp,
+ dshash_memhash,
+ dshash_memcpy
+};
+
+/*
+ * These are used for reporting memory context statistics of a process.
+ */
+
+/* Array to store the keys of MemoryStatsDsHash */
+static int *client_keys = NULL;
+
+/*
+ * Table to store pointers to DSA memory containing memory statistics and other
+ * metadata. There is one entry per client backend request, keyed by ProcNumber
+ * of the client obtained from client_keys array above.
+ */
+static dshash_table *MemoryStatsDsHash = NULL;
+
+/*
+ * Dsa area which stores the actual memory context
+ * statistics.
+ */
+static dsa_area *MemoryStatsDsaArea = NULL;
+
+static void memstats_dsa_cleanup(int ProcNumber);
+static void memstats_client_key_reset(int ProcNumber);
+static void PublishMemoryContext(MemoryStatsEntry *memcxt_info,
+ int curr_id, MemoryContext context,
+ List *path,
+ MemoryContextCounters stat,
+ int num_contexts);
/* ----------
* The max bytes for showing identifiers of MemoryContext.
+ * This is used by pg_get_backend_memory_context - view used for local backend.
* ----------
*/
#define MEMORY_CONTEXT_IDENT_DISPLAY_SIZE 1024
+/* Timeout in seconds */
+#define MEMORY_STATS_MAX_TIMEOUT 5
+
/*
* MemoryContextId
- * Used for storage of transient identifiers for
- * pg_get_backend_memory_contexts.
+ * Used for storage of transient identifiers for memory context reporting
*/
typedef struct MemoryContextId
{
@@ -63,6 +166,67 @@ int_list_to_array(const List *list)
return PointerGetDatum(result_array);
}
+/*
+ * context_type_to_string
+ * Returns a textual representation of a context type
+ *
+ * This should cover the same types as MemoryContextIsValid.
+ */
+static const char *
+context_type_to_string(NodeTag type)
+{
+ const char *context_type;
+
+ switch (type)
+ {
+ case T_AllocSetContext:
+ context_type = "AllocSet";
+ break;
+ case T_GenerationContext:
+ context_type = "Generation";
+ break;
+ case T_SlabContext:
+ context_type = "Slab";
+ break;
+ case T_BumpContext:
+ context_type = "Bump";
+ break;
+ default:
+ context_type = "???";
+ break;
+ }
+ return context_type;
+}
+
+/*
+ * compute_context_path
+ *
+ * Append the transient context_id of this context and each of its ancestors
+ * to a list, in order to compute a path.
+ */
+static List *
+compute_context_path(MemoryContext c, HTAB *context_id_lookup)
+{
+ bool found;
+ List *path = NIL;
+ MemoryContext cur_context;
+
+ for (cur_context = c; cur_context != NULL; cur_context = cur_context->parent)
+ {
+ MemoryContextId *cur_entry;
+
+ cur_entry = hash_search(context_id_lookup, &cur_context, HASH_FIND, &found);
+
+ if (!found)
+ {
+ elog(NOTICE, "hash table corrupted, can't construct path value");
+ return NIL;
+ }
+ path = lcons_int(cur_entry->context_id, path);
+ }
+ return path;
+}
+
/*
* PutMemoryContextsStatsTupleStore
* Add details for the given MemoryContext to 'tupstore'.
@@ -144,24 +308,7 @@ PutMemoryContextsStatsTupleStore(Tuplestorestate *tupstore,
else
nulls[1] = true;
- switch (context->type)
- {
- case T_AllocSetContext:
- type = "AllocSet";
- break;
- case T_GenerationContext:
- type = "Generation";
- break;
- case T_SlabContext:
- type = "Slab";
- break;
- case T_BumpContext:
- type = "Bump";
- break;
- default:
- type = "???";
- break;
- }
+ type = context_type_to_string(context->type);
values[2] = CStringGetTextDatum(type);
values[3] = Int32GetDatum(list_length(path)); /* level */
@@ -306,3 +453,841 @@ pg_log_backend_memory_contexts(PG_FUNCTION_ARGS)
PG_RETURN_BOOL(true);
}
+
+/*
+ * pg_get_process_memory_contexts
+ * Signal a backend or an auxiliary process to send its memory contexts,
+ * wait for the results and display them.
+ *
+ * By default, only superusers or users with ROLE_PG_READ_ALL_STATS are allowed
+ * to signal a process to return the memory contexts. Additional roles can be
+ * permitted with GRANT.
+ *
+ * On receipt of this signal, a backend or an auxiliary process sets the flag
+ * in the signal handler, which causes the next CHECK_FOR_INTERRUPTS()
+ * or process-specific interrupt handler to copy the memory context details
+ * to a dynamic shared memory space.
+ *
+ * We have defined a limit on DSA memory that could be allocated per process -
+ * if the process has more memory contexts than what can fit in the allocated
+ * size, the excess contexts are summarized and represented as cumulative total
+ * at the end of the buffer.
+ *
+ * After sending the signal, wait on a condition variable. The publishing
+ * backend, after copying the data to shared memory, sends a signal on that
+ * condition variable. There is one condition variable per client process.
+ * Once the condition variable is signalled, check if the latest memory context
+ * information is available and display.
+ *
+ * If the publishing backend does not respond before the condition variable
+ * times out, which is set to a predefined value MEMORY_STATS_MAX_TIMEOUT, give
+ * up and return NULL.
+ */
+Datum
+pg_get_process_memory_contexts(PG_FUNCTION_ARGS)
+{
+ int pid = PG_GETARG_INT32(0);
+ bool summary = PG_GETARG_BOOL(1);
+ PGPROC *proc;
+ ProcNumber client_procno = MyProcNumber;
+ ProcNumber server_procno;
+ bool proc_is_aux = false;
+ ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ MemoryStatsEntry *memcxt_info;
+ MemoryStatsDSHashEntry *entry;
+ bool found;
+ TimestampTz start_timestamp;
+
+ /*
+ * See if the process with given pid is a backend or an auxiliary process
+ * and remember the type for when we requery the process later.
+ */
+ proc = BackendPidGetProc(pid);
+ if (proc == NULL)
+ {
+ proc = AuxiliaryPidGetProc(pid);
+ proc_is_aux = true;
+ }
+
+ /*
+ * BackendPidGetProc() and AuxiliaryPidGetProc() return NULL if the pid
+ * isn't valid; this is however not a problem and leave with a WARNING.
+ * See comment in pg_log_backend_memory_contexts for a discussion on this.
+ */
+ if (proc == NULL)
+ {
+ ereport(WARNING,
+ errmsg("PID %d is not a PostgreSQL server process", pid));
+ PG_RETURN_NULL();
+ }
+
+ server_procno = GetNumberFromPGProc(proc);
+
+ /*
+ * First, check if the server process slot is occupied; if so, exit early.
+ * An occupied slot indicates another client backend is already requesting
+ * statistics from this server process. We should not set the slot yet,
+ * even if it appears empty, until we are prepared to signal the server
+ * process. This is because if the following DSA APIs throw an error, the
+ * server slot may not be reset, leading to it being incorrectly marked as
+ * occupied and blocking further memory context statistics queries to the
+ * same process.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[server_procno] != -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a DSA to allocate memory for copying memory contexts statistics.
+ * Allocate the memory in the DSA and send DSA pointer to the server
+ * process for storing the context statistics. If number of contexts
+ * exceed a predefined limit (1MB), a cumulative total is stored for such
+ * contexts.
+ *
+ * The DSA is created once for the lifetime of the server, and only
+ * attached in subsequent calls.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ /*
+ * The DSA pointers containing statistics for each client are stored in a
+ * dshash table. In addition to DSA pointer, each entry in this table also
+ * contains information about the statistics, condition variable for
+ * signalling between client and the server and miscellaneous data
+ * specific to a request. There is one entry per client request in the
+ * hash table.
+ */
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ /*
+ * Insert an entry for this client in DSHASH table the first time this
+ * function is called. This entry is deleted when the process exits in
+ * before_shmem_exit call.
+ *
+ * dshash_find_or_insert locks the entry to prevent the publisher from
+ * reading before client has updated the entry.
+ */
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &client_procno, &found);
+ if (!found)
+ {
+ entry->stats_written = false;
+ ConditionVariableInit(&entry->memcxt_cv);
+ }
+
+ /*
+ * Allocate 1MB of memory for the backend to publish its statistics on
+ * every call to this function. The memory is freed at the end of the
+ * function.
+ */
+ entry->memstats_dsa_pointer =
+ dsa_allocate0(MemoryStatsDsaArea, MEMORY_CONTEXT_REPORT_MAX_PER_BACKEND);
+
+ /*
+ * Specify whether a summary of statistics is requested, before signalling
+ * the server.
+ */
+ entry->summary = summary;
+
+ /*
+ * Indicate which server process statistics are being requested from. If
+ * this client times out before the last requested process can publish its
+ * statistics, it may send a new request to another server process. Since
+ * the previous server was notified, it might attempt to read the same
+ * client entry and respond incorrectly with its statistics. By storing
+ * the server ID in the client entry, we prevent any previously signalled
+ * server process from writing its statistics in the space meant for the
+ * newly requested process.
+ */
+ entry->target_server_id = pid;
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ /*
+ * Check if the publishing process slot is empty and store this clients
+ * key i.e its procNumber. This informs the publishing process that it is
+ * supposed to write statistics in the hash entry corresponding to this
+ * client.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[server_procno] == -1)
+ client_keys[server_procno] = MyProcNumber;
+ else
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ memstats_dsa_cleanup(client_procno);
+ ereport(WARNING,
+ errmsg("server process %d is processing previous request",
+ pid));
+ PG_RETURN_NULL();
+ }
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Send a signal to a PostgreSQL process, informing it we want it to
+ * produce information about its memory contexts.
+ */
+ if (SendProcSignal(pid, PROCSIG_GET_MEMORY_CONTEXT, server_procno) < 0)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ereport(WARNING,
+ errmsg("could not send signal to process %d: %m",
+ pid));
+ PG_RETURN_NULL();
+ }
+ start_timestamp = GetCurrentTimestamp();
+
+ while (1)
+ {
+ long elapsed_time;
+
+ INJECTION_POINT("memcontext-client-injection", NULL);
+
+ elapsed_time = TimestampDifferenceMilliseconds(start_timestamp,
+ GetCurrentTimestamp());
+ /* Return if we have already exceeded the timeout */
+ if (elapsed_time >= MEMORY_STATS_MAX_TIMEOUT * 1000)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics to PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Recheck the state of the backend before sleeping on the condition
+ * variable to ensure the process is still alive. Only check the
+ * relevant process type based on the earlier PID check.
+ */
+ if (proc_is_aux)
+ proc = AuxiliaryPidGetProc(pid);
+ else
+ proc = BackendPidGetProc(pid);
+
+ /*
+ * The target server process ending during memory context processing
+ * is not an error.
+ */
+ if (proc == NULL)
+ {
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(WARNING,
+ errmsg("PID %d is no longer a PostgreSQL server process",
+ pid));
+ PG_RETURN_NULL();
+ }
+
+ /*
+ * Wait for at most MEMORY_STATS_MAX_TIMEOUT seconds, if we have been
+ * woken up spuriously we subtract the time already slept. If no
+ * statistics are available within the allowed time then return no
+ * rows and emit a NOTICE. The timer is defined in milliseconds since
+ * that's what the condition variable sleep uses.
+ */
+ if (ConditionVariableTimedSleep(&entry->memcxt_cv,
+ (MEMORY_STATS_MAX_TIMEOUT * 1000) - elapsed_time,
+ WAIT_EVENT_MEM_CXT_PUBLISH))
+ {
+ /* Timeout has expired, return NULL */
+ memstats_dsa_cleanup(client_procno);
+ memstats_client_key_reset(server_procno);
+ ConditionVariableCancelSleep();
+ ereport(NOTICE,
+ errmsg("request for memory context statistics to PID %d timed out",
+ pid));
+ PG_RETURN_NULL();
+ }
+ entry = dshash_find_or_insert(MemoryStatsDsHash, &client_procno, &found);
+ Assert(found);
+
+ memcxt_info = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ /*
+ * We expect to come out of sleep when the requested process has
+ * finished publishing the statistics, verified using a boolean
+ * stats_written.
+ *
+ * Make sure that the statistics are actually written by checking that
+ * the name of the context is not NULL. This is done to ensure that
+ * the subsequent waits for statistics do not return spuriously if the
+ * previous call to the function ended in error and thus could not
+ * clear the stats_written flag.
+ */
+ if (entry->stats_written && memcxt_info[0].name[0] != '\0')
+ break;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ }
+
+ InitMaterializedSRF(fcinfo, 0);
+
+ /*
+ * Backend has finished publishing the stats, project them.
+ */
+#define PG_GET_PROCESS_MEMORY_CONTEXTS_COLS 11
+ for (int i = 0; i < entry->total_stats; i++)
+ {
+ ArrayType *path_array;
+ Datum values[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ bool nulls[PG_GET_PROCESS_MEMORY_CONTEXTS_COLS];
+ Datum *path_datum = NULL;
+
+ memset(values, 0, sizeof(values));
+ memset(nulls, 0, sizeof(nulls));
+
+ Assert(memcxt_info[i].name[0] != '\0');
+ values[0] = CStringGetTextDatum(memcxt_info[i].name);
+
+ if (memcxt_info[i].ident[0] != '\0')
+ values[1] = CStringGetTextDatum(memcxt_info[i].ident);
+ else
+ nulls[1] = true;
+
+ values[2] = CStringGetTextDatum(context_type_to_string(memcxt_info[i].type));
+
+ if (memcxt_info[i].levels != 0)
+ values[3] = Int32GetDatum(memcxt_info[i].levels);
+ else
+ nulls[3] = true;
+
+ if (memcxt_info[i].path[0] != 0)
+ {
+ int path_length;
+
+ path_length = memcxt_info[i].path_length;
+ path_datum = (Datum *) palloc(path_length * sizeof(Datum));
+
+ for (int j = 0; j < path_length; j++)
+ path_datum[j] = Int32GetDatum(memcxt_info[i].path[j]);
+ path_array = construct_array_builtin(path_datum,
+ path_length,
+ INT4OID);
+ values[4] = PointerGetDatum(path_array);
+ }
+ else
+ nulls[4] = true;
+
+ values[5] = Int64GetDatum(memcxt_info[i].totalspace);
+ values[6] = Int64GetDatum(memcxt_info[i].nblocks);
+ values[7] = Int64GetDatum(memcxt_info[i].freespace);
+ values[8] = Int64GetDatum(memcxt_info[i].freechunks);
+ values[9] = Int64GetDatum(memcxt_info[i].totalspace -
+ memcxt_info[i].freespace);
+ values[10] = Int32GetDatum(memcxt_info[i].num_agg_stats);
+
+ tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc,
+ values, nulls);
+ }
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ memstats_dsa_cleanup(client_procno);
+
+ ConditionVariableCancelSleep();
+
+ PG_RETURN_NULL();
+}
+
+/*
+ * This function is executed before returning from the
+ * pg_get_process_memory_contexts() function.
+ * It releases the DSA memory allocated for storing the memory statistics
+ * retrieved by that function. The caller must ensure that the LWLock for
+ * MemoryStatsDsHash is not already held.
+ */
+static void
+memstats_dsa_cleanup(int procNumber)
+{
+ MemoryStatsDSHashEntry *entry;
+
+ entry = dshash_find(MemoryStatsDsHash, &procNumber, true);
+
+ if (entry == NULL)
+ return;
+
+ Assert(MemoryStatsDsaArea != NULL);
+ if (DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ entry->memstats_dsa_pointer = InvalidDsaPointer;
+ entry->stats_written = false;
+ entry->target_server_id = 0;
+
+ dshash_release_lock(MemoryStatsDsHash, entry);
+}
+
+/*
+ * Remove this process from the publishing process'
+ * client key slot, if the stats publishing process has failed to do so.
+ */
+static void
+memstats_client_key_reset(int procNumber)
+{
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+
+ if (client_keys[procNumber] == MyProcNumber)
+ client_keys[procNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
+
+void
+MemoryContextKeysShmemInit(void)
+{
+ bool found;
+
+ client_keys = (int *)
+ ShmemInitStruct("MemoryContextKeys",
+ MemoryContextKeysShmemSize(), &found);
+
+ if (!found)
+ MemSet(client_keys, -1, MemoryContextKeysShmemSize());
+}
+
+Size
+MemoryContextKeysShmemSize(void)
+{
+ Size TotalProcs = 0;
+
+ TotalProcs = add_size(TotalProcs, NUM_AUXILIARY_PROCS);
+ TotalProcs = add_size(TotalProcs, MaxBackends);
+
+ return mul_size(TotalProcs, sizeof(int));
+}
+
+/*
+ * HandleGetMemoryContextInterrupt
+ * Handle receipt of an interrupt indicating a request to publish memory
+ * contexts statistics.
+ *
+ * All the actual work is deferred to ProcessGetMemoryContextInterrupt() as
+ * this cannot be performed in a signal handler.
+ */
+void
+HandleGetMemoryContextInterrupt(void)
+{
+ InterruptPending = true;
+ PublishMemoryContextPending = true;
+ /* latch will be set by procsignal_sigusr1_handler */
+}
+
+/*
+ * ProcessGetMemoryContextInterrupt
+ * Generate information about memory contexts used by the process.
+ *
+ * Performs a breadth first search on the memory context tree, thus parents
+ * statistics are reported before their children in the monitoring function
+ * output.
+ *
+ * Statistics for all the processes are shared via the same dynamic shared
+ * area. Individual statistics are tracked independently in per-process DSA
+ * pointers. These pointers are stored in a dshash table with key as requesting
+ * clients ProcNumber.
+ *
+ * We calculate maximum number of context's statistics that can be displayed
+ * using a pre-determined limit for memory available per process for this
+ * utility and maximum size of statistics for each context. The remaining
+ * context statistics if any are captured as a cumulative total at the end of
+ * individual context's statistics.
+ *
+ * If summary is true, we capture the level 1 and level 2 contexts statistics.
+ * For that we traverse the memory context tree recursively in depth first
+ * search manner to cover all the children of a parent context, to be able to
+ * display a cumulative total of memory consumption by a parent at level 2 and
+ * all its children.
+ */
+void
+ProcessGetMemoryContextInterrupt(void)
+{
+ List *contexts;
+ HASHCTL ctl;
+ HTAB *context_id_lookup;
+ int context_id = 0;
+ MemoryStatsEntry *meminfo;
+ MemoryContextCounters stat;
+ int num_individual_stats = 0;
+ bool found;
+ MemoryStatsDSHashEntry *entry;
+ int clientProcNumber;
+ MemoryContext memstats_ctx = NULL;
+ MemoryContext oldcontext = NULL;
+
+ PublishMemoryContextPending = false;
+
+ /*
+ * Avoid performing any shared memory operations in aborted transaction,
+ * the caller will timeout and return NULL.
+ */
+ if (IsAbortedTransactionBlockState())
+ return;
+
+ INJECTION_POINT("memcontext-server-wait", NULL);
+
+ /*
+ * Retrieve the client key for publishing statistics and reset it to -1,
+ * so other clients can request memory statistics from this process.
+ * Return if the client_key is -1, which means the requesting client has
+ * timed out.
+ */
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ if (client_keys[MyProcNumber] == -1)
+ {
+ LWLockRelease(MemContextReportingClientKeysLock);
+ return;
+ }
+ clientProcNumber = client_keys[MyProcNumber];
+ client_keys[MyProcNumber] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+
+ /*
+ * Create a new memory context which is not a part of TopMemoryContext
+ * tree. This context is used to allocate all memory in this function.
+ * This helps in keeping the memory allocation in this function to report
+ * memory consumption statistics separate. So that it does not affect the
+ * output of this function.
+ */
+ memstats_ctx = AllocSetContextCreate((MemoryContext) NULL,
+ "publish_memory_context_statistics",
+ ALLOCSET_SMALL_SIZES);
+ oldcontext = MemoryContextSwitchTo(memstats_ctx);
+
+ /*
+ * The hash table is used for constructing "path" column of the view,
+ * similar to its local backend counterpart.
+ */
+ ctl.keysize = sizeof(MemoryContext);
+ ctl.entrysize = sizeof(MemoryContextId);
+ ctl.hcxt = CurrentMemoryContext;
+
+ context_id_lookup = hash_create("pg_get_process_memory_contexts",
+ 256,
+ &ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+
+ /* List of contexts to process in the next round - start at the top. */
+ contexts = list_make1(TopMemoryContext);
+
+ /*
+ * The client process should have created the required DSA and DSHash
+ * table, so we can connect to these directly. The GetNamedXXX functions
+ * either return DSA/DSHash or error out (which we want to avoid as that
+ * would cause the current transaction to fail) so we don't need to check
+ * for NULL. An assertion is enough to catch in case this API should be
+ * changed at some point.
+ */
+ if (MemoryStatsDsaArea == NULL)
+ MemoryStatsDsaArea = GetNamedDSA("memory_context_statistics_dsa",
+ &found);
+
+ if (MemoryStatsDsHash == NULL)
+ MemoryStatsDsHash = GetNamedDSHash("memory_context_statistics_dshash",
+ &memctx_dsh_params, &found);
+
+ Assert(MemoryStatsDsaArea != NULL && MemoryStatsDsHash != NULL);
+
+ /*
+ * The entry lock is held by dshash_find to prevent the client backend to
+ * delete or read from the entry before we finish writing to it. Two
+ * different backends requesting statistics do not block each other.
+ */
+ entry = dshash_find(MemoryStatsDsHash, &clientProcNumber, true);
+
+ /*
+ * Check if the entry has been deleted due to calling process exiting, or
+ * if the caller has timed out waiting for us and have issued a request to
+ * another backend.
+ *
+ * Make sure that the client always deletes the entry after taking the
+ * required lock or this function may end up writing to unallocated
+ * memory.
+ */
+ if (entry == NULL || entry->target_server_id != MyProcPid)
+ {
+ dshash_release_lock(MemoryStatsDsHash, entry);
+
+ hash_destroy(context_id_lookup);
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+
+ return;
+ }
+
+ /* Should be allocated by a client backend that is requesting statistics */
+ Assert(entry->memstats_dsa_pointer != InvalidDsaPointer);
+ meminfo = (MemoryStatsEntry *)
+ dsa_get_address(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+
+ INJECTION_POINT("memcontext-server-injection", NULL);
+ if (entry->summary)
+ {
+ int cxt_id = 0;
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ /* Copy TopMemoryContext statistics to DSA */
+ memset(&stat, 0, sizeof(stat));
+ (*TopMemoryContext->methods->stats) (TopMemoryContext, NULL, NULL,
+ &stat, true);
+ path = lcons_int(1, path);
+ PublishMemoryContext(meminfo, cxt_id, TopMemoryContext, path, stat,
+ 1);
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &TopMemoryContext,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /* context id starts with 1 */
+ contextid_entry->context_id = cxt_id + 1;
+
+ /*
+ * Copy statistics for each of TopMemoryContext's children. This
+ * includes statistics of at most 100 children per node, with each
+ * child node limited to a depth of 100 in its subtree.
+ */
+ for (MemoryContext c = TopMemoryContext->firstchild; c != NULL;
+ c = c->nextchild)
+ {
+ MemoryContextCounters grand_totals;
+ int num_contexts = 0;
+
+ path = NIL;
+ memset(&grand_totals, 0, sizeof(grand_totals));
+
+ cxt_id++;
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &c, HASH_ENTER, &found);
+ Assert(!found);
+ contextid_entry->context_id = cxt_id + 1;
+
+ MemoryContextStatsCounter(c, &grand_totals, &num_contexts);
+
+ path = compute_context_path(c, context_id_lookup);
+
+ PublishMemoryContext(meminfo, cxt_id, c, path,
+ grand_totals, num_contexts);
+ }
+ entry->total_stats = cxt_id + 1;
+ }
+ else
+ {
+ foreach_ptr(MemoryContextData, cur, contexts)
+ {
+ List *path = NIL;
+ MemoryContextId *contextid_entry;
+
+ contextid_entry = (MemoryContextId *) hash_search(context_id_lookup,
+ &cur,
+ HASH_ENTER, &found);
+ Assert(!found);
+
+ /*
+ * context id starts with 1
+ */
+ contextid_entry->context_id = context_id + 1;
+
+ /*
+ * Figure out the transient context_id of this context and each of
+ * its ancestors, to compute a path for this context.
+ */
+ path = compute_context_path(cur, context_id_lookup);
+
+ /* Examine the context stats */
+ memset(&stat, 0, sizeof(stat));
+ (*cur->methods->stats) (cur, NULL, NULL, &stat, true);
+
+ /* Account for saving one statistics slot for cumulative reporting */
+ if (context_id < (MAX_MEMORY_CONTEXT_STATS_NUM - 1))
+ {
+ /* Copy statistics to DSA memory */
+ PublishMemoryContext(meminfo, context_id, cur, path, stat, 1);
+ }
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].totalspace += stat.totalspace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].nblocks += stat.nblocks;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freespace += stat.freespace;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].freechunks += stat.freechunks;
+ }
+
+ /*
+ * DSA max limit per process is reached, write aggregate of the
+ * remaining statistics.
+ *
+ * We can store contexts from 0 to max_stats - 1. When context_id
+ * is greater than max_stats, we stop reporting individual
+ * statistics when context_id equals max_stats - 2. As we use
+ * max_stats - 1 array slot for reporting cumulative statistics or
+ * "Remaining Totals".
+ */
+ if (context_id == (MAX_MEMORY_CONTEXT_STATS_NUM - 2))
+ {
+ int namelen = strlen("Remaining Totals");
+
+ num_individual_stats = context_id + 1;
+ strlcpy(meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].name,
+ "Remaining Totals", namelen + 1);
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].ident[0] = '\0';
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].path[0] = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].type = 0;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].levels = 0;
+ }
+ context_id++;
+
+ for (MemoryContext c = cur->firstchild; c != NULL; c = c->nextchild)
+ contexts = lappend(contexts, c);
+ }
+
+ /*
+ * Check if there are aggregated statistics or not in the result set.
+ * Statistics are individually reported when context_id <= max_stats,
+ * only if context_id > max_stats will there be aggregates.
+ */
+ if (context_id <= MAX_MEMORY_CONTEXT_STATS_NUM)
+ {
+ entry->total_stats = context_id;
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats = 1;
+ }
+
+ /*
+ * The number of contexts exceeded the space available, so report the
+ * number of aggregated memory contexts
+ */
+ else
+ {
+ meminfo[MAX_MEMORY_CONTEXT_STATS_NUM - 1].num_agg_stats =
+ context_id - num_individual_stats;
+
+ /*
+ * Total stats equals num_individual_stats + 1 record for
+ * cumulative statistics.
+ */
+ entry->total_stats = num_individual_stats + 1;
+ }
+ }
+
+ entry->stats_written = true;
+ /* Notify waiting client backend and return */
+ ConditionVariableSignal(&entry->memcxt_cv);
+ dshash_release_lock(MemoryStatsDsHash, entry);
+ hash_destroy(context_id_lookup);
+
+ MemoryContextSwitchTo(oldcontext);
+ MemoryContextDelete(memstats_ctx);
+}
+
+/*
+ * PublishMemoryContext
+ *
+ * Copy the memory context statistics of a single context to a DSA memory
+ */
+static void
+PublishMemoryContext(MemoryStatsEntry *memcxt_info, int curr_id,
+ MemoryContext context, List *path,
+ MemoryContextCounters stat, int num_contexts)
+{
+ char *ident = unconstify(char *, context->ident);
+ char *name = unconstify(char *, context->name);
+
+ /*
+ * To be consistent with logging output, we label dynahash contexts with
+ * just the hash table name as with MemoryContextStatsPrint().
+ */
+ if (context->ident && strcmp(context->name, "dynahash") == 0)
+ {
+ name = unconstify(char *, context->ident);
+ ident = NULL;
+ }
+
+ if (name != NULL)
+ {
+ int namelen = strlen(name);
+
+ if (namelen >= MEMORY_CONTEXT_NAME_SHMEM_SIZE)
+ namelen = pg_mbcliplen(name, namelen,
+ MEMORY_CONTEXT_NAME_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].name, name, namelen + 1);
+ }
+ else
+ /* Clearing the array */
+ memcxt_info[curr_id].name[0] = '\0';
+
+ /* Trim and copy the identifier if it is not set to NULL */
+ if (ident != NULL)
+ {
+ int idlen = strlen(ident);
+
+ /*
+ * Some identifiers such as SQL query string can be very long,
+ * truncate oversize identifiers.
+ */
+ if (idlen >= MEMORY_CONTEXT_IDENT_SHMEM_SIZE)
+ idlen = pg_mbcliplen(ident, idlen,
+ MEMORY_CONTEXT_IDENT_SHMEM_SIZE - 1);
+
+ strlcpy(memcxt_info[curr_id].ident, ident, idlen + 1);
+ }
+ else
+ memcxt_info[curr_id].ident[0] = '\0';
+
+ /* Store the path */
+ if (path == NIL)
+ memcxt_info[curr_id].path[0] = 0;
+ else
+ {
+ int levels = Min(list_length(path), MAX_PATH_DISPLAY_LENGTH);
+
+ memcxt_info[curr_id].path_length = levels;
+ memcxt_info[curr_id].levels = list_length(path);
+
+ foreach_int(i, path)
+ {
+ memcxt_info[curr_id].path[foreach_current_index(i)] = i;
+ if (--levels == 0)
+ break;
+ }
+ }
+ memcxt_info[curr_id].type = context->type;
+ memcxt_info[curr_id].totalspace = stat.totalspace;
+ memcxt_info[curr_id].nblocks = stat.nblocks;
+ memcxt_info[curr_id].freespace = stat.freespace;
+ memcxt_info[curr_id].freechunks = stat.freechunks;
+ memcxt_info[curr_id].num_agg_stats = num_contexts;
+}
+
+void
+AtProcExit_memstats_cleanup(int code, Datum arg)
+{
+ int idx = MyProcNumber;
+ MemoryStatsDSHashEntry *entry;
+
+ if (MemoryStatsDsHash != NULL)
+ {
+ entry = dshash_find(MemoryStatsDsHash, &idx, true);
+
+ if (entry != NULL)
+ {
+ if (MemoryStatsDsaArea != NULL &&
+ DsaPointerIsValid(entry->memstats_dsa_pointer))
+ dsa_free(MemoryStatsDsaArea, entry->memstats_dsa_pointer);
+ dshash_delete_entry(MemoryStatsDsHash, entry);
+ }
+ }
+ LWLockAcquire(MemContextReportingClientKeysLock, LW_EXCLUSIVE);
+ client_keys[idx] = -1;
+ LWLockRelease(MemContextReportingClientKeysLock);
+}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index 36ad708b360..e246fc7ce1f 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -39,6 +39,7 @@ volatile sig_atomic_t TransactionTimeoutPending = false;
volatile sig_atomic_t IdleSessionTimeoutPending = false;
volatile sig_atomic_t ProcSignalBarrierPending = false;
volatile sig_atomic_t LogMemoryContextPending = false;
+volatile sig_atomic_t PublishMemoryContextPending = false;
volatile sig_atomic_t IdleStatsUpdateTimeoutPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index b59e08605cc..ba5d3bf0d7a 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -669,6 +669,13 @@ BaseInit(void)
* drop ephemeral slots, which in turn triggers stats reporting.
*/
ReplicationSlotInitialize();
+
+ /*
+ * The before shmem exit callback frees the DSA memory occupied by the
+ * latest memory context statistics that could be published by this proc
+ * if requested.
+ */
+ before_shmem_exit(AtProcExit_memstats_cleanup, 0);
}
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 073bdb35d2a..9780a5b090c 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -1011,6 +1011,37 @@ MemoryContextStatsInternal(MemoryContext context, int level,
}
}
+
+/*
+ * MemoryContextStatsCounter
+ *
+ * Accumulate statistics counts into *totals. totals should not be NULL.
+ * This involves a non-recursive tree traversal.
+ */
+void
+MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts)
+{
+ int ichild = 1;
+
+ Assert(totals != NULL && num_contexts != NULL);
+ context->methods->stats(context, NULL, NULL, totals, false);
+
+ for (MemoryContext curr = context->firstchild;
+ curr != NULL;
+ curr = MemoryContextTraverseNext(curr, context))
+ {
+ curr->methods->stats(curr, NULL, NULL, totals, false);
+ ichild++;
+ }
+
+ /*
+ * Add the count of all the children contexts which are traversed
+ * including the parent.
+ */
+ *num_contexts = ichild;
+}
+
/*
* MemoryContextStatsPrint
* Print callback used by MemoryContextStatsInternal
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index dac40992cbc..3667bc2bf41 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -8641,6 +8641,16 @@
prorettype => 'bool', proargtypes => 'int4',
prosrc => 'pg_log_backend_memory_contexts' },
+# publishing memory contexts of the specified postgres process
+{ oid => '2173', descr => 'publish memory contexts of the specified process',
+ proname => 'pg_get_process_memory_contexts', provolatile => 'v',
+ prorows => '100', proretset => 't', proparallel => 'u',
+ prorettype => 'record', proargtypes => 'int4 bool',
+ proallargtypes => '{int4,bool,text,text,text,int4,_int4,int8,int8,int8,int8,int8,int4}',
+ proargmodes => '{i,i,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{pid, summary, name, ident, type, level, path, total_bytes, total_nblocks, free_bytes, free_chunks, used_bytes, num_agg_contexts}',
+ prosrc => 'pg_get_process_memory_contexts' },
+
# non-persistent series generator
{ oid => '1066', descr => 'non-persistent series generator',
proname => 'generate_series', prorows => '1000',
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index f16f35659b9..38e1e55a9a6 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -96,6 +96,7 @@ extern PGDLLIMPORT volatile sig_atomic_t IdleSessionTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcSignalBarrierPending;
extern PGDLLIMPORT volatile sig_atomic_t LogMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t IdleStatsUpdateTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t PublishMemoryContextPending;
extern PGDLLIMPORT volatile sig_atomic_t CheckClientConnectionPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/storage/lwlocklist.h b/src/include/storage/lwlocklist.h
index e94ebce95b9..6dd80689bbb 100644
--- a/src/include/storage/lwlocklist.h
+++ b/src/include/storage/lwlocklist.h
@@ -87,6 +87,7 @@ PG_LWLOCK(52, SerialControl)
PG_LWLOCK(53, AioWorkerSubmissionQueue)
PG_LWLOCK(54, WaitLSN)
PG_LWLOCK(55, LogicalDecodingControl)
+PG_LWLOCK(56, MemContextReportingClientKeys)
/*
* There also exist several built-in LWLock tranches. As with the predefined
diff --git a/src/include/storage/procsignal.h b/src/include/storage/procsignal.h
index 348fba53a93..00ccfcb22a4 100644
--- a/src/include/storage/procsignal.h
+++ b/src/include/storage/procsignal.h
@@ -35,6 +35,7 @@ typedef enum
PROCSIG_WALSND_INIT_STOPPING, /* ask walsenders to prepare for shutdown */
PROCSIG_BARRIER, /* global barrier interrupt */
PROCSIG_LOG_MEMORY_CONTEXT, /* ask backend to log the memory contexts */
+ PROCSIG_GET_MEMORY_CONTEXT, /* ask backend to send the memory contexts */
PROCSIG_PARALLEL_APPLY_MESSAGE, /* Message from parallel apply workers */
PROCSIG_RECOVERY_CONFLICT, /* backend is blocking recovery, check
* PGPROC->pendingRecoveryConflicts for the
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 11ab1717a16..40099036308 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -319,4 +319,11 @@ pg_memory_is_all_zeros(const void *ptr, size_t len)
return true;
}
+extern void ProcessGetMemoryContextInterrupt(void);
+extern void HandleGetMemoryContextInterrupt(void);
+extern void MemoryContextKeysShmemInit(void);
+extern Size MemoryContextKeysShmemSize(void);
+extern void MemoryContextStatsCounter(MemoryContext context, MemoryContextCounters *totals,
+ int *num_contexts);
+extern void AtProcExit_memstats_cleanup(int code, Datum arg);
#endif /* MEMUTILS_H */
diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out
index 3dd63fd88ed..c84e5adce16 100644
--- a/src/test/regress/expected/sysviews.out
+++ b/src/test/regress/expected/sysviews.out
@@ -233,3 +233,22 @@ select * from pg_timezone_abbrevs where abbrev = 'LMT';
LMT | @ 7 hours 52 mins 58 secs ago | f
(1 row)
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
+NOTICE: (AllocSet,TopMemoryContext,)
+NOTICE: (AllocSet,TopMemoryContext,)
diff --git a/src/test/regress/sql/sysviews.sql b/src/test/regress/sql/sysviews.sql
index 004f9a70e00..d34d2b6211c 100644
--- a/src/test/regress/sql/sysviews.sql
+++ b/src/test/regress/sql/sysviews.sql
@@ -101,3 +101,21 @@ select count(distinct utc_offset) >= 24 as ok from pg_timezone_abbrevs;
-- One specific case we can check without much fear of breakage
-- is the historical local-mean-time value used for America/Los_Angeles.
select * from pg_timezone_abbrevs where abbrev = 'LMT';
+
+DO $$
+DECLARE
+ bg_writer_pid int;
+ r RECORD;
+BEGIN
+ SELECT pid from pg_stat_activity where backend_type='background writer'
+ INTO bg_writer_pid;
+
+ select type, name, ident
+ from pg_get_process_memory_contexts(bg_writer_pid, false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+ select type, name, ident
+ from pg_get_process_memory_contexts(pg_backend_pid(), false)
+ where path = '{1}' into r;
+ RAISE NOTICE '%', r;
+END $$;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 77e3c04144e..a0370d1bb80 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -1710,6 +1710,8 @@ MemoryContextData
MemoryContextId
MemoryContextMethodID
MemoryContextMethods
+MemoryStatsEntry
+MemoryStatsDSHashEntry
MemoryStatsPrintFunc
MergeAction
MergeActionState
--
2.34.1
^ permalink raw reply [nested|flat] 65+ messages in thread
end of thread, other threads:[~2026-03-05 22:31 UTC | newest]
Thread overview: 65+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-08-01 14:20 Re: Allow COPY's 'text' format to output a header Daniel Verite <[email protected]>
2018-08-01 15:18 ` Cynthia Shang <[email protected]>
2018-08-01 18:36 ` Simon Muller <[email protected]>
2018-08-02 12:11 ` Daniel Verite <[email protected]>
2018-08-02 15:07 ` Cynthia Shang <[email protected]>
2018-08-02 19:30 ` Simon Muller <[email protected]>
2018-08-03 16:30 ` Cynthia Shang <[email protected]>
2018-08-06 14:34 ` Stephen Frost <[email protected]>
2018-08-08 18:57 ` Simon Muller <[email protected]>
2018-08-09 14:37 ` Cynthia Shang <[email protected]>
2018-08-17 04:39 ` Michael Paquier <[email protected]>
2019-05-03 14:24 [PATCH v5] make \d pg_toast.foo show its indices Justin Pryzby <[email protected]>
2019-05-03 14:24 [PATCH v2 1/2] make \d pg_toast.foo show its indices Justin Pryzby <[email protected]>
2019-05-03 14:24 [PATCH v7] make \d pg_toast.foo show its indices Justin Pryzby <[email protected]>
2019-05-03 14:24 [PATCH v3] make \d pg_toast.foo show its indices Justin Pryzby <[email protected]>
2019-05-03 14:24 [PATCH v5] make \d pg_toast.foo show its indices Justin Pryzby <[email protected]>
2019-05-03 14:24 [PATCH v3] make \d pg_toast.foo show its indices Justin Pryzby <[email protected]>
2021-12-17 15:35 [PATCH 4/4] Move the double-plus "Size" columns to the right Justin Pryzby <[email protected]>
2021-12-17 15:35 [PATCH 4/4] Move the double-plus "Size" columns to the right Justin Pryzby <[email protected]>
2021-12-17 15:35 [PATCH 4/4] Move the double-plus "Size" columns to the right Justin Pryzby <[email protected]>
2021-12-17 15:35 [PATCH v4 4/4] Move the double-plus "Size" columns to the right Justin Pryzby <[email protected]>
2021-12-17 15:35 [PATCH 4/4] Move the double-plus "Size" columns to the right Justin Pryzby <[email protected]>
2021-12-17 15:35 [PATCH 4/4] Move the double-plus "Size" columns to the right Justin Pryzby <[email protected]>
2021-12-17 15:35 [PATCH 4/4] Move the double-plus "Size" columns to the right Justin Pryzby <[email protected]>
2021-12-17 15:35 [PATCH v8 4/4] Move the double-plus "Size" columns to the right Justin Pryzby <[email protected]>
2021-12-17 15:35 [PATCH 4/4] Move the double-plus "Size" columns to the right Justin Pryzby <[email protected]>
2025-04-30 10:43 Re: Enhancing Memory Context Statistics Reporting Daniel Gustafsson <[email protected]>
2025-07-11 15:31 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-07-29 13:40 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-08-08 09:26 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-08-12 00:34 ` Re: Enhancing Memory Context Statistics Reporting torikoshia <[email protected]>
2025-08-13 22:35 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-08-18 13:32 ` Re: Enhancing Memory Context Statistics Reporting torikoshia <[email protected]>
2025-08-19 21:42 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-10-07 00:51 ` Re: Enhancing Memory Context Statistics Reporting torikoshia <[email protected]>
2025-10-09 08:43 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-10-14 09:00 ` Re: Enhancing Memory Context Statistics Reporting torikoshia <[email protected]>
2025-10-28 05:36 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-11-07 22:55 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-11-20 09:56 ` Re: Enhancing Memory Context Statistics Reporting Daniel Gustafsson <[email protected]>
2025-11-25 07:20 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-11-28 09:22 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-12-08 01:26 ` Re: Enhancing Memory Context Statistics Reporting torikoshia <[email protected]>
2025-12-12 09:46 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-12-08 09:43 ` Re: Enhancing Memory Context Statistics Reporting Daniel Gustafsson <[email protected]>
2025-12-12 11:04 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-12-18 19:54 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2025-12-29 08:56 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2026-01-09 11:22 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2026-01-13 19:31 ` Re: Enhancing Memory Context Statistics Reporting Robert Haas <[email protected]>
2026-01-13 20:26 ` Re: Enhancing Memory Context Statistics Reporting Andres Freund <[email protected]>
2026-01-13 21:11 ` Re: Enhancing Memory Context Statistics Reporting Robert Haas <[email protected]>
2026-01-13 21:47 ` Re: Enhancing Memory Context Statistics Reporting Andres Freund <[email protected]>
2026-01-14 13:26 ` Re: Enhancing Memory Context Statistics Reporting Robert Haas <[email protected]>
2026-01-14 13:32 ` Re: Enhancing Memory Context Statistics Reporting Andres Freund <[email protected]>
2026-01-19 17:31 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2026-01-19 18:02 ` Re: Enhancing Memory Context Statistics Reporting Robert Haas <[email protected]>
2026-02-02 12:28 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2026-02-10 00:03 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2026-02-10 04:50 ` Re: Enhancing Memory Context Statistics Reporting Chao Li <[email protected]>
2026-02-15 12:40 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2026-02-20 23:13 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2026-02-23 09:58 ` Re: Enhancing Memory Context Statistics Reporting Daniel Gustafsson <[email protected]>
2026-02-24 11:57 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
2026-03-05 22:31 ` Re: Enhancing Memory Context Statistics Reporting Rahila Syed <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox