Content-Type: multipart/alternative;
 boundary="------------l06pmdO3lduMXIrjJWVop05z"
Message-ID: <4358bd85-f6b4-4da6-9909-74428fe3c8f7@gmail.com>
Date: Mon, 27 Apr 2026 21:00:00 +0300
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Startup process deadlock: WaitForProcSignalBarriers vs aux
 process
To: Masahiko Sawada <sawada.mshk@gmail.com>,
 Andres Freund <andres@anarazel.de>
Cc: Matthias van de Meent <boekewurm+postgres@gmail.com>,
 Thomas Munro <thomas.munro@gmail.com>,
 PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
 Heikki Linnakangas <hlinnaka@iki.fi>
References: 
 <CAEze2WgAJmWReDN7Chtba8Er2YBvKCoa0KVN25-1evnTrHsLyA@mail.gmail.com>
 <ojnnvj553eber32njg5nphybcf6zvb3bzexfkh3obylsll67pc@jk7jmtzeb66k>
 <CAD21AoBj+zKvgw_Q8gjr4YbKccW_uMe3OFQ5+KT246FHUuNXSQ@mail.gmail.com>
Content-Language: en-US
From: Alexander Lakhin <exclusion@gmail.com>
In-Reply-To: 
 <CAD21AoBj+zKvgw_Q8gjr4YbKccW_uMe3OFQ5+KT246FHUuNXSQ@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/4358bd85-f6b4-4da6-9909-74428fe3c8f7%40gmail.com>
Precedence: bulk

This is a multi-part message in MIME format.
--------------l06pmdO3lduMXIrjJWVop05z
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hello Sawada-san,

24.04.2026 20:52, Masahiko Sawada wrote:
> Right. The postmaster blocks all signals before starting child process
> as the following comment explains:
>
>       /*
>        * We start postmaster children with signals blocked.  This allows them to
>        * install their own handlers before unblocking, to avoid races where they
>        * might run the postmaster's handler and miss an important control
>        * signal. With more analysis this could potentially be relaxed.
>        */
>       sigprocmask(SIG_SETMASK, &BlockSig, &save_mask);
>
> Investigating the issue, I found there is a race condition between the
> procsignal initialization and emitting signal barrier that could be
> the cause of this issue. Imagine the following scenario:
>
> 1. In ProcSignalInit(), the checkpointer initializes its
> slot->pss_barrierGeneration with the global generation.
> 2. In EmitProcSignalBarrier(), the startup checks the checkpointer's
> procsignal slot but it skips emitting the signal as slot->pss_pid is
> still 0. It can happen even though the checkpointer holds a spinlock
> on its slot during the initialization because the first pid check is
> done without a spinlock acquisition.
> 3. The checkpointer sets its pid to slot->pss_pid and releases the spin lock.
> 4. In WaitForProcSignalBarrier(), the startup checks the
> checkpointer's procsignal slot that has already initialized the
> pss_barrierGeneration, and waits for it to be updated. However, the
> checkpointer never updates its barrier generation as it doesn't get
> the signal.

Thank you for the investigation and explanation of the issue!

I've been puzzled by a buildfarm failure [1] with such symptoms for a while
and even reproduced it locally once, but couldn't gather more information
that time. But now that you have described the scenario, I can easily
reproduce the same test failure with:
--- a/src/backend/storage/ipc/procsignal.c
+++ b/src/backend/storage/ipc/procsignal.c
@@ -206,6 +206,7 @@ ProcSignalInit(const uint8 *cancel_key, int cancel_key_len)
         if (cancel_key_len > 0)
                 memcpy(slot->pss_cancel_key, cancel_key, cancel_key_len);
         slot->pss_cancel_key_len = cancel_key_len;
+pg_usleep(10000);
         pg_atomic_write_u32(&slot->pss_pid, MyProcPid);

just running `meson test test_oat_hooks_*/regress` with the test multiplied x30:
26/30 test_oat_hooks_28 - postgresql:test_oat_hooks_28/regress         OK 1.28s   2 subtests passed
27/30 test_oat_hooks_30 - postgresql:test_oat_hooks_30/regress         OK 1.25s   2 subtests passed
28/30 test_oat_hooks_2 - postgresql:test_oat_hooks_2/regress           ERROR 62.49s   exit status 2

2026-04-27 17:34:44.290 UTC postmaster[1578102] LOG:  starting PostgreSQL 19devel on x86_64-linux, compiled by 
gcc-16.0.1, 64-bit
2026-04-27 17:34:44.290 UTC postmaster[1578102] LOG:  listening on Unix socket "/tmp/pg_regress-QdhMPt/.s.PGSQL.40086"
2026-04-27 17:34:44.302 UTC startup[1578114] LOG:  database system was shut down at 2026-04-27 17:34:44 UTC
2026-04-27 17:34:44.325 UTC dead-end client backend[1578133] [unknown] FATAL:  the database system is starting up
...
2026-04-27 17:34:49.274 UTC dead-end client backend[1578643] [unknown] FATAL:  the database system is starting up
2026-04-27 17:34:49.308 UTC startup[1578114] LOG:  still waiting for backend with PID 1578110 to accept ProcSignalBarrier
2026-04-27 17:34:49.325 UTC dead-end client backend[1578645] [unknown] FATAL:  the database system is starting up
...
2026-04-27 17:35:44.332 UTC dead-end client backend[1582376] [unknown] FATAL:  the database system is starting up
2026-04-27 17:35:44.351 UTC startup[1578114] LOG:  still waiting for backend with PID 1578110 to accept ProcSignalBarrier
2026-04-27 17:35:44.383 UTC dead-end client backend[1582379] [unknown] FATAL:  the database system is starting up


[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&dt=2026-03-10%2013%3A58%3A55

Best regards,
Alexander
--------------l06pmdO3lduMXIrjJWVop05z
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <div class="moz-cite-prefix">Hello Sawada-san,<br>
      <br>
      24.04.2026 20:52, Masahiko Sawada wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAD21AoBj+zKvgw_Q8gjr4YbKccW_uMe3OFQ5+KT246FHUuNXSQ@mail.gmail.com">
      <pre wrap="" class="moz-quote-pre">
Right. The postmaster blocks all signals before starting child process
as the following comment explains:

     /*
      * We start postmaster children with signals blocked.  This allows them to
      * install their own handlers before unblocking, to avoid races where they
      * might run the postmaster's handler and miss an important control
      * signal. With more analysis this could potentially be relaxed.
      */
     sigprocmask(SIG_SETMASK, &amp;BlockSig, &amp;save_mask);

Investigating the issue, I found there is a race condition between the
procsignal initialization and emitting signal barrier that could be
the cause of this issue. Imagine the following scenario:

1. In ProcSignalInit(), the checkpointer initializes its
slot-&gt;pss_barrierGeneration with the global generation.
2. In EmitProcSignalBarrier(), the startup checks the checkpointer's
procsignal slot but it skips emitting the signal as slot-&gt;pss_pid is
still 0. It can happen even though the checkpointer holds a spinlock
on its slot during the initialization because the first pid check is
done without a spinlock acquisition.
3. The checkpointer sets its pid to slot-&gt;pss_pid and releases the spin lock.
4. In WaitForProcSignalBarrier(), the startup checks the
checkpointer's procsignal slot that has already initialized the
pss_barrierGeneration, and waits for it to be updated. However, the
checkpointer never updates its barrier generation as it doesn't get
the signal.
</pre>
    </blockquote>
    <br>
    Thank you for the investigation and explanation of the issue!<br>
    <br>
    I've been puzzled by a buildfarm failure [1] with such symptoms for
    a while<br>
    and even reproduced it locally once, but couldn't gather more
    information<br>
    that time. But now that you have described the scenario, I can
    easily<br>
    reproduce the same test failure with:<br>
    --- a/src/backend/storage/ipc/procsignal.c<br>
    +++ b/src/backend/storage/ipc/procsignal.c<br>
    @@ -206,6 +206,7 @@ ProcSignalInit(const uint8 *cancel_key, int
    cancel_key_len)<br>
            if (cancel_key_len &gt; 0)<br>
                    memcpy(slot-&gt;pss_cancel_key, cancel_key,
    cancel_key_len);<br>
            slot-&gt;pss_cancel_key_len = cancel_key_len;<br>
    +pg_usleep(10000);<br>
            pg_atomic_write_u32(&amp;slot-&gt;pss_pid, MyProcPid);<br>
    <br>
    just running `meson test test_oat_hooks_*/regress` with the test
    multiplied x30:<br>
    26/30 test_oat_hooks_28 -
    postgresql:test_oat_hooks_28/regress         OK               
    1.28s   2 subtests passed<br>
    27/30 test_oat_hooks_30 -
    postgresql:test_oat_hooks_30/regress         OK               
    1.25s   2 subtests passed<br>
    28/30 test_oat_hooks_2 -
    postgresql:test_oat_hooks_2/regress           ERROR           
    62.49s   exit status 2<br>
    <br>
    2026-04-27 17:34:44.290 UTC postmaster[1578102] LOG:  starting
    PostgreSQL 19devel on x86_64-linux, compiled by gcc-16.0.1, 64-bit<br>
    2026-04-27 17:34:44.290 UTC postmaster[1578102] LOG:  listening on
    Unix socket "/tmp/pg_regress-QdhMPt/.s.PGSQL.40086"<br>
    2026-04-27 17:34:44.302 UTC startup[1578114] LOG:  database system
    was shut down at 2026-04-27 17:34:44 UTC<br>
    2026-04-27 17:34:44.325 UTC dead-end client backend[1578133]
    [unknown] FATAL:  the database system is starting up<br>
    ...<br>
    2026-04-27 17:34:49.274 UTC dead-end client backend[1578643]
    [unknown] FATAL:  the database system is starting up<br>
    2026-04-27 17:34:49.308 UTC startup[1578114] LOG:  still waiting for
    backend with PID 1578110 to accept ProcSignalBarrier<br>
    2026-04-27 17:34:49.325 UTC dead-end client backend[1578645]
    [unknown] FATAL:  the database system is starting up<br>
    ...<br>
    2026-04-27 17:35:44.332 UTC dead-end client backend[1582376]
    [unknown] FATAL:  the database system is starting up<br>
    2026-04-27 17:35:44.351 UTC startup[1578114] LOG:  still waiting for
    backend with PID 1578110 to accept ProcSignalBarrier<br>
    2026-04-27 17:35:44.383 UTC dead-end client backend[1582379]
    [unknown] FATAL:  the database system is starting up<br>
    <br>
    <br>
    [1]
<a class="moz-txt-link-freetext" href="https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&amp;dt=2026-03-10%2013%3A58%3A55">https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=flaviventris&amp;dt=2026-03-10%2013%3A58%3A55</a><br>
    <br>
    Best regards,<br>
    Alexander<br>
  </body>
</html>

--------------l06pmdO3lduMXIrjJWVop05z--