MIME-Version: 1.0
References: 
 <CAEze2WgAJmWReDN7Chtba8Er2YBvKCoa0KVN25-1evnTrHsLyA@mail.gmail.com>
 <ojnnvj553eber32njg5nphybcf6zvb3bzexfkh3obylsll67pc@jk7jmtzeb66k>
 <CAD21AoBj+zKvgw_Q8gjr4YbKccW_uMe3OFQ5+KT246FHUuNXSQ@mail.gmail.com>
 <4358bd85-f6b4-4da6-9909-74428fe3c8f7@gmail.com>
 <CAD21AoBDjU4fi+QgaC7RimqrDoUi9Ma+oBP10i0Je13NPSYcuw@mail.gmail.com>
In-Reply-To: 
 <CAD21AoBDjU4fi+QgaC7RimqrDoUi9Ma+oBP10i0Je13NPSYcuw@mail.gmail.com>
From: Matthias van de Meent <boekewurm+postgres@gmail.com>
Date: Wed, 29 Apr 2026 12:49:24 +0200
Message-ID: 
 <CAEze2WhhTnSLpjGJWGupbxkTp_JdNP6v0mNgpqhi_YkXJa=m6A@mail.gmail.com>
Subject: Re: Startup process deadlock: WaitForProcSignalBarriers vs aux
 process
To: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: Alexander Lakhin <exclusion@gmail.com>,
 Andres Freund <andres@anarazel.de>,
	Thomas Munro <thomas.munro@gmail.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
 Heikki Linnakangas <hlinnaka@iki.fi>
Content-Type: multipart/mixed; boundary="000000000000aab6890650971ae7"
Archived-At: 
 <https://www.postgresql.org/message-id/CAEze2WhhTnSLpjGJWGupbxkTp_JdNP6v0mNgpqhi_YkXJa%3Dm6A%40mail.gmail.com>
Precedence: bulk

--000000000000aab6890650971ae7
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Wed, 22 Apr 2026 at 21:05, Andres Freund <andres@anarazel.de> wrote:
>
> Hi,
>
> On 2026-04-22 13:21:02 +0200, Matthias van de Meent wrote:
> > If the PSB is emitted (and signaled to checkpointer) before the
> > checkpointer has registered its SIGUSR1 handler, then the checkpointer
> > won't receive the notice to check its procsignal slots, it won't
> > notice the updated procsignal flags, and it won't process the PSB; not
> > until it receives a new SIGUSR1.
> >
> > Signals are sent to all processes that have their procsignal pss_pid
> > set, which is true for every process which has called ProcSignalInit,
> > which for the checkpointer (like other aux processes) happens in
> > AuxiliaryProcessMainCommon. However, checkpointer (also like other aux
> > processes) calls AuxiliaryProcessMainCommon before registering its
> > signal handlers, creating a small window in time where signals are
> > sent, but not handled.
>
> Hm. Have we confirmed this happens?
>
> CheckpointerMain() is called with all signals masked, so it should be ok =
for
> the signal handler to only be set up after AuxiliaryProcessMainCommon(), =
as
> long as it happens before [...]

Yeah, that was a misidentification of the exact race that caused the issue.

On Tue, 28 Apr 2026 at 21:28, Masahiko Sawada <sawada.mshk@gmail.com> wrote=
:
>
> On Mon, Apr 27, 2026 at 11:00=E2=80=AFAM Alexander Lakhin <exclusion@gmai=
l.com> wrote:
> >
> > Hello Sawada-san,
> >
> > 24.04.2026 20:52, Masahiko Sawada wrote:
> >
> > Right. The postmaster blocks all signals before starting child process
> > as the following comment explains:
> >
> >      /*
> >       * We start postmaster children with signals blocked.  This allows=
 them to
> >       * install their own handlers before unblocking, to avoid races wh=
ere they
> >       * might run the postmaster's handler and miss an important contro=
l
> >       * signal. With more analysis this could potentially be relaxed.
> >       */
> >      sigprocmask(SIG_SETMASK, &BlockSig, &save_mask);
> >
> > Investigating the issue, I found there is a race condition between the
> > procsignal initialization and emitting signal barrier that could be
> > the cause of this issue. Imagine the following scenario:

Ah, that'd be it indeed. Thanks!

> I've attached a patch to address the issue. I haven't verified it
> across all versions yet, but I suspect it exists in the stable
> branches as well. Previously, the issue rarely occurred because
> EmitProcSignalBarrier() was only used for smgr invalidation. However,
> now that we use signal barriers for online wal_level changes and
> checksum status updates, this race condition is likely to be
> encountered more frequently.

Yes, I think the boot process with the xlog_logical_info barrier is
more likely to hit this issue; as indicated by two known detected
cases in various CI jobs; though it could also be that the lockup of
the new barrier is just exceptionally bad for system stability.

As for the patches:
v1-0001 -- LGTM.

0001 (upthread): LGTM, but I'd also suggest to add some code to make
sure that we're actually receiving procsignals by the time we
initialize the Logical/Checksum subsystems that need to process shared
state changes by responding to procsignals; as attached. smgr's
procsignal doesn't really depend on shared memory state, so I've kept
that out of my patch.


Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)

--000000000000aab6890650971ae7
Content-Type: application/octet-stream;
	name="v1-0001-Assert-ProcSignal-is-initialized-before-its-depen.patch"
Content-Disposition: attachment;
	filename="v1-0001-Assert-ProcSignal-is-initialized-before-its-depen.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_mojxm31t0>
X-Attachment-Id: f_mojxm31t0

RnJvbSA4YTFkYzE4YmJjZjExYTJlYjM2Y2ZkM2RiYjI5MDk3NmQ4NzI4NGQxIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBNYXR0aGlhcyB2YW4gZGUgTWVlbnQgPGJvZWtld3VybStwb3N0
Z3Jlc0BnbWFpbC5jb20+CkRhdGU6IFdlZCwgMjkgQXByIDIwMjYgMTI6MTA6NDQgKzAyMDAKU3Vi
amVjdDogW1BBVENIIHYxXSBBc3NlcnQgUHJvY1NpZ25hbCBpcyBpbml0aWFsaXplZCBiZWZvcmUg
aXRzIGRlcGVuZGVudHMKCi0tLQogc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGxvZy5jICAg
ICAgICAgICAgfCAxICsKIHNyYy9iYWNrZW5kL3JlcGxpY2F0aW9uL2xvZ2ljYWwvbG9naWNhbGN0
bC5jIHwgMSArCiBzcmMvYmFja2VuZC9zdG9yYWdlL2lwYy9wcm9jc2lnbmFsLmMgICAgICAgICB8
IDggKysrKysrKysKIHNyYy9pbmNsdWRlL3N0b3JhZ2UvcHJvY3NpZ25hbC5oICAgICAgICAgICAg
IHwgNSArKysrKwogNCBmaWxlcyBjaGFuZ2VkLCAxNSBpbnNlcnRpb25zKCspCgpkaWZmIC0tZ2l0
IGEvc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGxvZy5jIGIvc3JjL2JhY2tlbmQvYWNjZXNz
L3RyYW5zYW0veGxvZy5jCmluZGV4IGUzOWFmNzljMDNiLi42M2U4NGIwMGNlYyAxMDA2NDQKLS0t
IGEvc3JjL2JhY2tlbmQvYWNjZXNzL3RyYW5zYW0veGxvZy5jCisrKyBiL3NyYy9iYWNrZW5kL2Fj
Y2Vzcy90cmFuc2FtL3hsb2cuYwpAQCAtNDk2MCw2ICs0OTYwLDcgQEAgU2V0RGF0YUNoZWNrc3Vt
c09mZih2b2lkKQogdm9pZAogSW5pdExvY2FsRGF0YUNoZWNrc3VtU3RhdGUodm9pZCkKIHsKKwlB
c3NlcnQoUHJvY1NpZ25hbElzSW5pdGlhbGl6ZWQoKSk7CiAJU3BpbkxvY2tBY3F1aXJlKCZYTG9n
Q3RsLT5pbmZvX2xjayk7CiAJU2V0TG9jYWxEYXRhQ2hlY2tzdW1TdGF0ZShYTG9nQ3RsLT5kYXRh
X2NoZWNrc3VtX3ZlcnNpb24pOwogCVNwaW5Mb2NrUmVsZWFzZSgmWExvZ0N0bC0+aW5mb19sY2sp
OwpkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvcmVwbGljYXRpb24vbG9naWNhbC9sb2dpY2FsY3Rs
LmMgYi9zcmMvYmFja2VuZC9yZXBsaWNhdGlvbi9sb2dpY2FsL2xvZ2ljYWxjdGwuYwppbmRleCA3
MmY2OGVjNThlZi4uODAzMDhiNjE5YTQgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL3JlcGxpY2F0
aW9uL2xvZ2ljYWwvbG9naWNhbGN0bC5jCisrKyBiL3NyYy9iYWNrZW5kL3JlcGxpY2F0aW9uL2xv
Z2ljYWwvbG9naWNhbGN0bC5jCkBAIC0xNzMsNiArMTczLDcgQEAgdXBkYXRlX3hsb2dfbG9naWNh
bF9pbmZvKHZvaWQpCiB2b2lkCiBJbml0aWFsaXplUHJvY2Vzc1hMb2dMb2dpY2FsSW5mbyh2b2lk
KQogeworCUFzc2VydChQcm9jU2lnbmFsSXNJbml0aWFsaXplZCgpKTsKIAl1cGRhdGVfeGxvZ19s
b2dpY2FsX2luZm8oKTsKIH0KIApkaWZmIC0tZ2l0IGEvc3JjL2JhY2tlbmQvc3RvcmFnZS9pcGMv
cHJvY3NpZ25hbC5jIGIvc3JjL2JhY2tlbmQvc3RvcmFnZS9pcGMvcHJvY3NpZ25hbC5jCmluZGV4
IGIwNjgxY2EwYWUyLi43MWEwYjI1ZTQ5ZSAxMDA2NDQKLS0tIGEvc3JjL2JhY2tlbmQvc3RvcmFn
ZS9pcGMvcHJvY3NpZ25hbC5jCisrKyBiL3NyYy9iYWNrZW5kL3N0b3JhZ2UvaXBjL3Byb2NzaWdu
YWwuYwpAQCAtMjMyLDYgKzIzMiwxNCBAQCBQcm9jU2lnbmFsSW5pdChjb25zdCB1aW50OCAqY2Fu
Y2VsX2tleSwgaW50IGNhbmNlbF9rZXlfbGVuKQogCW9uX3NobWVtX2V4aXQoQ2xlYW51cFByb2NT
aWduYWxTdGF0ZSwgKERhdHVtKSAwKTsKIH0KIAorI2lmZGVmIFVTRV9BU1NFUlRfQ0hFQ0tJTkcK
K2Jvb2wKK1Byb2NTaWduYWxJc0luaXRpYWxpemVkKHZvaWQpCit7CisJcmV0dXJuIE15UHJvY1Np
Z25hbFNsb3QgIT0gTlVMTDsKK30KKyNlbmRpZgorCiAvKgogICogQ2xlYW51cFByb2NTaWduYWxT
dGF0ZQogICoJCVJlbW92ZSBjdXJyZW50IHByb2Nlc3MgZnJvbSBQcm9jU2lnbmFsIG1lY2hhbmlz
bQpkaWZmIC0tZ2l0IGEvc3JjL2luY2x1ZGUvc3RvcmFnZS9wcm9jc2lnbmFsLmggYi9zcmMvaW5j
bHVkZS9zdG9yYWdlL3Byb2NzaWduYWwuaAppbmRleCBhYWExNThiZmQ2Ni4uMWQyMjkwYzY5NzUg
MTAwNjQ0Ci0tLSBhL3NyYy9pbmNsdWRlL3N0b3JhZ2UvcHJvY3NpZ25hbC5oCisrKyBiL3NyYy9p
bmNsdWRlL3N0b3JhZ2UvcHJvY3NpZ25hbC5oCkBAIC04Nyw0ICs4Nyw5IEBAIHR5cGVkZWYgc3Ry
dWN0IFByb2NTaWduYWxIZWFkZXIgUHJvY1NpZ25hbEhlYWRlcjsKIGV4dGVybiBQR0RMTElNUE9S
VCBQcm9jU2lnbmFsSGVhZGVyICpQcm9jU2lnbmFsOwogI2VuZGlmCiAKKyNpZmRlZiBVU0VfQVNT
RVJUX0NIRUNLSU5HCitleHRlcm4gYm9vbCBQcm9jU2lnbmFsSXNJbml0aWFsaXplZCh2b2lkKTsK
KyNlbmRpZgorCisKICNlbmRpZgkJCQkJCQkvKiBQUk9DU0lHTkFMX0ggKi8KLS0gCjIuNTAuMSAo
QXBwbGUgR2l0LTE1NSkKCg==
--000000000000aab6890650971ae7--