Content-Type: multipart/alternative;
 boundary="------------sND80aAPB7VNlvPmWXwSXhlc"
Message-ID: <58b5d6bf-7dca-41e5-8000-e12981205ada@gmail.com>
Date: Tue, 12 May 2026 23:00:00 +0300
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: BUG #18158: Assert in pgstat_report_stat() fails when a backend
 shutting down with stats pending
To: pgsql-bugs@lists.postgresql.org
References: <18158-88f667028dbc7e7b@postgresql.org>
Content-Language: en-US
From: Alexander Lakhin <exclusion@gmail.com>
In-Reply-To: <18158-88f667028dbc7e7b@postgresql.org>
Archived-At: 
 <https://www.postgresql.org/message-id/58b5d6bf-7dca-41e5-8000-e12981205ada%40gmail.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------sND80aAPB7VNlvPmWXwSXhlc
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hello hackers,

16.10.2023 12:00, PG Bug reporting form wrote:
> The following bug has been logged on the website:
>
> Bug reference:      18158
> Logged by:          Alexander Lakhin
> Email address:exclusion@gmail.com
> PostgreSQL version: 16.0
> Operating system:   Ubuntu 22.04
> Description:
> ...
> With the following modification in pgstat_flush_pending_entries():
> +if (nowait && (rand() % 10 == 0))
> +        did_flush = false;
> +else
> +{
>           did_flush = kind_info->flush_pending_cb(entry_ref, nowait);
> +}
>
> the issue reproduced easily:
> make -s check -C src/test/recovery/ PROVE_TESTS="t/012_subtransactions.pl"
> grep TRAP -r src/test/recovery/tmp_check/log
>
> # +++ tap check in src/test/recovery +++
> t/012_subtransactions.pl .. ok
> All tests successful.
> Files=1, Tests=12,  3 wallclock secs ( 0.01 usr  0.00 sys +  0.19 cusr  0.27
> csys =  0.47 CPU)
> Result: PASS
> src/test/recovery/tmp_check/log/012_subtransactions_primary.log:
> TRAP: failed Assert("!pgStatLocal.shmem->is_shutdown"), File: "pgstat.c",
> Line: 616, PID: 2410126

It looks like skink produced this failure yesterday [1]:
241/316 recovery - postgresql:recovery/027_stream_regress ERROR           4374.08s   exit status 1
...
stderr:
#   Failed test 'check contents of pg_stat_statements on regression database'
#   at /home/bf/bf-build/skink/REL_17_STABLE/pgsql/src/test/recovery/t/027_stream_regress.pl line 177.
#          got: 'CREATE|f
# SELECT|t'
#     expected: 'CREATE|t
# DELETE|t
# INSERT|t
# SELECT|t
# UPDATE|t'
# Looks like you failed 1 test of 9.

  pgsql.build/testrun/recovery/027_stream_regress/log/027_stream_regress_primary.log

2026-05-11 14:09:34.397 CEST [4064132][walsender][40/0:0] LOG: released physical replication slot "standby_1"
TRAP: failed Assert("!pgStatLocal.shmem->is_shutdown"), File: "../pgsql/src/backend/utils/activity/pgstat.c", Line: 612, 
PID: 4064132
postgres: primary: walsender bf [local] streaming 0/15B4FC98(ExceptionalCondition+0x5f) [0x45ae268]
postgres: primary: walsender bf [local] streaming 0/15B4FC98(pgstat_report_stat+0x14d) [0x4492fe2]
postgres: primary: walsender bf [local] streaming 0/15B4FC98(+0x49315c) [0x449315c]
postgres: primary: walsender bf [local] streaming 0/15B4FC98(shmem_exit+0x78) [0x4448f00]
postgres: primary: walsender bf [local] streaming 0/15B4FC98(+0x449020) [0x4449020]
postgres: primary: walsender bf [local] streaming 0/15B4FC98(proc_exit+0x22) [0x44490c1]
postgres: primary: walsender bf [local] streaming 0/15B4FC98(+0x3f5dbd) [0x43f5dbd]
...
2026-05-11 14:09:34.629 CEST [4063470][postmaster][:0] LOG:  server process (PID 4064132) was terminated by signal 6: 
Aborted
2026-05-11 14:09:34.629 CEST [4063470][postmaster][:0] DETAIL: Failed process was running: START_REPLICATION SLOT 
"standby_1" 0/3000000 TIMELINE 1
2026-05-11 14:09:34.631 CEST [4063470][postmaster][:0] LOG: terminating any other active server processes

And perhaps (I can't find the full log now) one year ago [2]:
227/305 postgresql:recovery / recovery/027_stream_regress ERROR           3364.50s   exit status 1

[01:19:39.781](0.108s) not ok 9 - check contents of pg_stat_statements on regression database
[01:19:39.781](0.000s) #   Failed test 'check contents of pg_stat_statements on regression database'
#   at /home/bf/bf-build/skink/REL_17_STABLE/pgsql/src/test/recovery/t/027_stream_regress.pl line 173.
[01:19:39.781](0.000s) #          got: 'CREATE|f
# SELECT|t'
#     expected: 'CREATE|t
# DELETE|t
# INSERT|t
# SELECT|t
# UPDATE|t'

I've reproduced such failures with the above modification applied, just
running:
for i in {1..20}; do PROVE_TESTS="t/027*" make -s check -C src/test/recovery/ || break; done

Reproduced on REL_17_STABLE and REL_16_STABLE (starting from dd8008e8e,
which updated 027_stream_regress.pl).

Not reproduced on REL_18_STABLE after 87a6690cc (coming from [3]).

[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2026-05-11%2010%3A25%3A22
[2] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2025-03-18%2023%3A39%3A44
[3] https://www.postgresql.org/message-id/yegnemsijlrhocsfgs7gs7irnczjgkom6fmk2a5u2b66pbvzwi%40ph2h3wppcvdy

Best regards,
Alexander
--------------sND80aAPB7VNlvPmWXwSXhlc
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">Hello hackers,<br>
      <br>
      16.10.2023 12:00, PG Bug reporting form wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:18158-88f667028dbc7e7b@postgresql.org">
      <pre wrap="" class="moz-quote-pre">The following bug has been logged on the website:

Bug reference:      18158
Logged by:          Alexander Lakhin
Email address:      <a class="moz-txt-link-abbreviated" href="mailto:exclusion@gmail.com">exclusion@gmail.com</a>
PostgreSQL version: 16.0
Operating system:   Ubuntu 22.04
Description:        
...
With the following modification in pgstat_flush_pending_entries():
+if (nowait &amp;&amp; (rand() % 10 == 0))
+        did_flush = false;
+else
+{
         did_flush = kind_info-&gt;flush_pending_cb(entry_ref, nowait);
+}

the issue reproduced easily:
make -s check -C src/test/recovery/ PROVE_TESTS="t/012_subtransactions.pl"
grep TRAP -r src/test/recovery/tmp_check/log

# +++ tap check in src/test/recovery +++
t/012_subtransactions.pl .. ok    
All tests successful.
Files=1, Tests=12,  3 wallclock secs ( 0.01 usr  0.00 sys +  0.19 cusr  0.27
csys =  0.47 CPU)
Result: PASS
src/test/recovery/tmp_check/log/012_subtransactions_primary.log:
TRAP: failed Assert("!pgStatLocal.shmem-&gt;is_shutdown"), File: "pgstat.c",
Line: 616, PID: 2410126
</pre>
    </blockquote>
    <br>
    It looks like skink produced this failure yesterday [1]:<br>
    241/316 recovery -
    postgresql:recovery/027_stream_regress                              
    ERROR           4374.08s   exit status 1<br>
    ...<br>
    stderr:<br>
    #   Failed test 'check contents of pg_stat_statements on regression
    database'<br>
    #   at
/home/bf/bf-build/skink/REL_17_STABLE/pgsql/src/test/recovery/t/027_stream_regress.pl
    line 177.<br>
    #          got: 'CREATE|f<br>
    # SELECT|t'<br>
    #     expected: 'CREATE|t<br>
    # DELETE|t<br>
    # INSERT|t<br>
    # SELECT|t<br>
    # UPDATE|t'<br>
    # Looks like you failed 1 test of 9.<br>
    <br>
 pgsql.build/testrun/recovery/027_stream_regress/log/027_stream_regress_primary.log<br>
     <br>
    2026-05-11 14:09:34.397 CEST [4064132][walsender][40/0:0] LOG: 
    released physical replication slot "standby_1"<br>
    TRAP: failed Assert("!pgStatLocal.shmem-&gt;is_shutdown"), File:
    "../pgsql/src/backend/utils/activity/pgstat.c", Line: 612, PID:
    4064132<br>
    postgres: primary: walsender bf [local] streaming
    0/15B4FC98(ExceptionalCondition+0x5f) [0x45ae268]<br>
    postgres: primary: walsender bf [local] streaming
    0/15B4FC98(pgstat_report_stat+0x14d) [0x4492fe2]<br>
    postgres: primary: walsender bf [local] streaming
    0/15B4FC98(+0x49315c) [0x449315c]<br>
    postgres: primary: walsender bf [local] streaming
    0/15B4FC98(shmem_exit+0x78) [0x4448f00]<br>
    postgres: primary: walsender bf [local] streaming
    0/15B4FC98(+0x449020) [0x4449020]<br>
    postgres: primary: walsender bf [local] streaming
    0/15B4FC98(proc_exit+0x22) [0x44490c1]<br>
    postgres: primary: walsender bf [local] streaming
    0/15B4FC98(+0x3f5dbd) [0x43f5dbd]<br>
    ...<br>
    2026-05-11 14:09:34.629 CEST [4063470][postmaster][:0] LOG:  server
    process (PID 4064132) was terminated by signal 6: Aborted<br>
    2026-05-11 14:09:34.629 CEST [4063470][postmaster][:0] DETAIL: 
    Failed process was running: START_REPLICATION SLOT "standby_1"
    0/3000000 TIMELINE 1<br>
    2026-05-11 14:09:34.631 CEST [4063470][postmaster][:0] LOG: 
    terminating any other active server processes<br>
    <br>
    And perhaps (I can't find the full log now) one year ago [2]:<br>
    227/305 postgresql:recovery /
    recovery/027_stream_regress                              
    ERROR           3364.50s   exit status 1<br>
    <br>
    [01:19:39.781](0.108s) not ok 9 - check contents of
    pg_stat_statements on regression database<br>
    [01:19:39.781](0.000s) #   Failed test 'check contents of
    pg_stat_statements on regression database'<br>
    #   at
/home/bf/bf-build/skink/REL_17_STABLE/pgsql/src/test/recovery/t/027_stream_regress.pl
    line 173.<br>
    [01:19:39.781](0.000s) #          got: 'CREATE|f<br>
    # SELECT|t'<br>
    #     expected: 'CREATE|t<br>
    # DELETE|t<br>
    # INSERT|t<br>
    # SELECT|t<br>
    # UPDATE|t'<br>
    <br>
    I've reproduced such failures with the above modification applied,
    just<br>
    running:<br>
    for i in {1..20}; do PROVE_TESTS="t/027*" make -s check -C
    src/test/recovery/ || break; done<br>
    <br>
    Reproduced on REL_17_STABLE and REL_16_STABLE (starting from
    dd8008e8e,<br>
    which updated 027_stream_regress.pl).<br>
    <br>
    Not reproduced on REL_18_STABLE after 87a6690cc (coming from [3]).<br>
    <br>
    [1]
<a class="moz-txt-link-freetext" href="https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&amp;dt=2026-05-11%2010%3A25%3A22">https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&amp;dt=2026-05-11%2010%3A25%3A22</a><br>
    [2]
<a class="moz-txt-link-freetext" href="https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&amp;dt=2025-03-18%2023%3A39%3A44">https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&amp;dt=2025-03-18%2023%3A39%3A44</a><br>
    [3]
<a class="moz-txt-link-freetext" href="https://www.postgresql.org/message-id/yegnemsijlrhocsfgs7gs7irnczjgkom6fmk2a5u2b66pbvzwi%40ph2h3wppcvdy">https://www.postgresql.org/message-id/yegnemsijlrhocsfgs7gs7irnczjgkom6fmk2a5u2b66pbvzwi%40ph2h3wppcvdy</a><br>
    <br>
    Best regards,<br>
    Alexander<br>
  </body>
</html>

--------------sND80aAPB7VNlvPmWXwSXhlc--