MIME-Version: 1.0
References: 
 <CAEze2WgAJmWReDN7Chtba8Er2YBvKCoa0KVN25-1evnTrHsLyA@mail.gmail.com>
 <ojnnvj553eber32njg5nphybcf6zvb3bzexfkh3obylsll67pc@jk7jmtzeb66k>
 <CAD21AoBj+zKvgw_Q8gjr4YbKccW_uMe3OFQ5+KT246FHUuNXSQ@mail.gmail.com>
 <4358bd85-f6b4-4da6-9909-74428fe3c8f7@gmail.com>
In-Reply-To: <4358bd85-f6b4-4da6-9909-74428fe3c8f7@gmail.com>
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Tue, 28 Apr 2026 12:27:59 -0700
Message-ID: 
 <CAD21AoBDjU4fi+QgaC7RimqrDoUi9Ma+oBP10i0Je13NPSYcuw@mail.gmail.com>
Subject: Re: Startup process deadlock: WaitForProcSignalBarriers vs aux
 process
To: Alexander Lakhin <exclusion@gmail.com>
Cc: Andres Freund <andres@anarazel.de>,
	Matthias van de Meent <boekewurm+postgres@gmail.com>,
 Thomas Munro <thomas.munro@gmail.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>,
 Heikki Linnakangas <hlinnaka@iki.fi>
Content-Type: multipart/mixed; boundary="000000000000dc803506508a3cd0"
Archived-At: 
 <https://www.postgresql.org/message-id/CAD21AoBDjU4fi%2BQgaC7RimqrDoUi9Ma%2BoBP10i0Je13NPSYcuw%40mail.gmail.com>
Precedence: bulk

--000000000000dc803506508a3cd0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, Apr 27, 2026 at 11:00=E2=80=AFAM Alexander Lakhin <exclusion@gmail.=
com> wrote:
>
> Hello Sawada-san,
>
> 24.04.2026 20:52, Masahiko Sawada wrote:
>
> Right. The postmaster blocks all signals before starting child process
> as the following comment explains:
>
>      /*
>       * We start postmaster children with signals blocked.  This allows t=
hem to
>       * install their own handlers before unblocking, to avoid races wher=
e they
>       * might run the postmaster's handler and miss an important control
>       * signal. With more analysis this could potentially be relaxed.
>       */
>      sigprocmask(SIG_SETMASK, &BlockSig, &save_mask);
>
> Investigating the issue, I found there is a race condition between the
> procsignal initialization and emitting signal barrier that could be
> the cause of this issue. Imagine the following scenario:
>
> 1. In ProcSignalInit(), the checkpointer initializes its
> slot->pss_barrierGeneration with the global generation.
> 2. In EmitProcSignalBarrier(), the startup checks the checkpointer's
> procsignal slot but it skips emitting the signal as slot->pss_pid is
> still 0. It can happen even though the checkpointer holds a spinlock
> on its slot during the initialization because the first pid check is
> done without a spinlock acquisition.
> 3. The checkpointer sets its pid to slot->pss_pid and releases the spin l=
ock.
> 4. In WaitForProcSignalBarrier(), the startup checks the
> checkpointer's procsignal slot that has already initialized the
> pss_barrierGeneration, and waits for it to be updated. However, the
> checkpointer never updates its barrier generation as it doesn't get
> the signal.
>
>
> Thank you for the investigation and explanation of the issue!
>
> I've been puzzled by a buildfarm failure [1] with such symptoms for a whi=
le
> and even reproduced it locally once, but couldn't gather more information
> that time. But now that you have described the scenario, I can easily
> reproduce the same test failure with:
> --- a/src/backend/storage/ipc/procsignal.c
> +++ b/src/backend/storage/ipc/procsignal.c
> @@ -206,6 +206,7 @@ ProcSignalInit(const uint8 *cancel_key, int cancel_ke=
y_len)
>         if (cancel_key_len > 0)
>                 memcpy(slot->pss_cancel_key, cancel_key, cancel_key_len);
>         slot->pss_cancel_key_len =3D cancel_key_len;
> +pg_usleep(10000);
>         pg_atomic_write_u32(&slot->pss_pid, MyProcPid);

Thank you for testing this.

I've attached a patch to address the issue. I haven't verified it
across all versions yet, but I suspect it exists in the stable
branches as well. Previously, the issue rarely occurred because
EmitProcSignalBarrier() was only used for smgr invalidation. However,
now that we use signal barriers for online wal_level changes and
checksum status updates, this race condition is likely to be
encountered more frequently.

Regards,

--=20
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

--000000000000dc803506508a3cd0
Content-Type: text/x-patch; charset="US-ASCII";
	name="v1-0001-Fix-race-between-ProcSignalInit-and-EmitProcSigna.patch"
Content-Disposition: attachment;
	filename="v1-0001-Fix-race-between-ProcSignalInit-and-EmitProcSigna.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_moj0psmi0>
X-Attachment-Id: f_moj0psmi0

RnJvbSA4ZWQ3MmUxYmM3NDhmOTlmYmY4YjEwM2FlNWJkNGNmMzk1Y2I1NGVmIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBNYXNhaGlrbyBTYXdhZGEgPHNhd2FkYS5tc2hrQGdtYWlsLmNv
bT4KRGF0ZTogVHVlLCAyOCBBcHIgMjAyNiAxMjoyMToyMSAtMDcwMApTdWJqZWN0OiBbUEFUQ0gg
djFdIEZpeCByYWNlIGJldHdlZW4gUHJvY1NpZ25hbEluaXQoKSBhbmQKIEVtaXRQcm9jU2lnbmFs
QmFycmllcigpLgoKUHJvY1NpZ25hbEluaXQoKSByZWFkIHRoZSBnbG9iYWwgYmFycmllciBnZW5l
cmF0aW9uIGJlZm9yZSBwdWJsaXNoaW5nCml0cyBQSUQgaW50byBwc3NfcGlkLiBBIGNvbmN1cnJl
bnQgRW1pdFByb2NTaWduYWxCYXJyaWVyKCkgaXRlcmF0ZXMKdGhlIFByb2NTaWduYWwgc2xvdHMg
YW5kIHNraXBzIGFueSB3aG9zZSBwc3NfcGlkIGlzIHN0aWxsIHplcm8sIG9uIHRoZQphc3N1bXB0
aW9uIHRoYXQgc3VjaCBhIHNsb3Qgd2lsbCBwaWNrIHVwIHRoZSBuZXcgZ2VuZXJhdGlvbiB3aGVu
IGl0CmxhdGVyIHJlYWRzIHBzaF9iYXJyaWVyR2VuZXJhdGlvbi4gQnV0IGJlY2F1c2UgdGhlIGpv
aW5pbmcgYmFja2VuZCBoYWQKYWxyZWFkeSByZWFkIHRoZSAob2xkZXIpIGdsb2JhbCBnZW5lcmF0
aW9uIHVuZGVyIGl0cyBzbG90J3Mgc3BpbmxvY2ssCml0IHdvdWxkIHN0b3JlIGEgc3RhbGUgdmFs
dWUgaW50byBwc3NfYmFycmllckdlbmVyYXRpb24gYW5kIG5ldmVyCmFic29yYiB0aGUganVzdC1l
bWl0dGVkIGJhcnJpZXIsIHJlc3VsdGluZyB0aGF0CldhaXRGb3JQcm9jU2lnbmFsQmFycmllcigp
IGRpZG4ndCBjb21wbGV0ZS4KClB1Ymxpc2ggcHNzX3BpZCBiZWZvcmUgcmVhZGluZyBwc2hfYmFy
cmllckdlbmVyYXRpb24sIHdpdGggYSBtZW1vcnkKYmFycmllciBpbiBiZXR3ZWVuIHNvIHRoYXQg
dGhlIHN0b3JlIGlzIGdsb2JhbGx5IHZpc2libGUgZmlyc3QuCUEKY29uY3VycmVudCBFbWl0UHJv
Y1NpZ25hbEJhcnJpZXIoKSB0aGVuIGVpdGhlciBvYnNlcnZlcyB0aGUgcHVibGlzaGVkClBJRCBh
bmQgc2lnbmFscyB0aGlzIHNsb3QsIG9yIGNvbXBsZXRlcyBpdHMgZ2VuZXJhdGlvbiBpbmNyZW1l
bnQKYmVmb3JlIHdlIGxvYWQgaXQuCgpEaXNjdXNzaW9uOiBodHRwczovL3Bvc3Rnci5lcy9tL0NB
RXplMldnQUptV1JlRE43Q2h0YmE4RXIyWUJ2S0NvYTBLVk4yNS0xZXZuVHJIc0x5QUBtYWlsLmdt
YWlsLmNvbQpCYWNrcGF0Y2gtdGhyb3VnaDoKLS0tCiBzcmMvYmFja2VuZC9zdG9yYWdlL2lwYy9w
cm9jc2lnbmFsLmMgfCAxMSArKysrKysrKysrLQogMSBmaWxlIGNoYW5nZWQsIDEwIGluc2VydGlv
bnMoKyksIDEgZGVsZXRpb24oLSkKCmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC9zdG9yYWdlL2lw
Yy9wcm9jc2lnbmFsLmMgYi9zcmMvYmFja2VuZC9zdG9yYWdlL2lwYy9wcm9jc2lnbmFsLmMKaW5k
ZXggMjY0ZTRjMjJjYTYuLmIwNjgxY2EwYWUyIDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC9zdG9y
YWdlL2lwYy9wcm9jc2lnbmFsLmMKKysrIGIvc3JjL2JhY2tlbmQvc3RvcmFnZS9pcGMvcHJvY3Np
Z25hbC5jCkBAIC0xODgsNiArMTg4LDE2IEBAIFByb2NTaWduYWxJbml0KGNvbnN0IHVpbnQ4ICpj
YW5jZWxfa2V5LCBpbnQgY2FuY2VsX2tleV9sZW4pCiAJLyogQ2xlYXIgb3V0IGFueSBsZWZ0b3Zl
ciBzaWduYWwgcmVhc29ucyAqLwogCU1lbVNldChzbG90LT5wc3Nfc2lnbmFsRmxhZ3MsIDAsIE5V
TV9QUk9DU0lHTkFMUyAqIHNpemVvZihzaWdfYXRvbWljX3QpKTsKIAorCS8qCisJICogUHVibGlz
aCB0aGUgUElEIGJlZm9yZSByZWFkaW5nIHRoZSBnbG9iYWwgYmFycmllciBnZW5lcmF0aW9uIHRv
IGVuc3VyZQorCSAqIHRoYXQgRW1pdFByb2NTaWduYWxCYXJyaWVyKCkgZG9lc24ndCBza2lwIHVz
IHdoaWxlIHdlIGFyZSBncmFiYmluZyBhbgorCSAqIG9sZGVyIGdlbmVyYXRpb24uIFdlIG5lZWQg
YSBtZW1vcnkgYmFycmllciBoZXJlIHRvIG1ha2Ugc3VyZSB0aGF0IHRoZQorCSAqIHVwZGF0ZSBv
ZiBwc3NfcGlkIGlzIGdsb2JhbGx5IHZpc2libGUgYmVmb3JlIHRoZSBsb2FkIG9mIHRoZSBnbG9i
YWwKKwkgKiBiYXJyaWVyIGdlbmVyYXRpb24gZXhlY3V0ZXMuCisJICovCisJcGdfYXRvbWljX3dy
aXRlX3UzMigmc2xvdC0+cHNzX3BpZCwgTXlQcm9jUGlkKTsKKwlwZ19tZW1vcnlfYmFycmllcigp
OworCiAJLyoKIAkgKiBJbml0aWFsaXplIGJhcnJpZXIgc3RhdGUuIFNpbmNlIHdlJ3JlIGEgYnJh
bmQtbmV3IHByb2Nlc3MsIHRoZXJlCiAJICogc2hvdWxkbid0IGJlIGFueSBsZWZ0b3ZlciBiYWNr
ZW5kLXByaXZhdGUgc3RhdGUgdGhhdCBuZWVkcyB0byBiZQpAQCAtMjA3LDcgKzIxNyw2IEBAIFBy
b2NTaWduYWxJbml0KGNvbnN0IHVpbnQ4ICpjYW5jZWxfa2V5LCBpbnQgY2FuY2VsX2tleV9sZW4p
CiAJaWYgKGNhbmNlbF9rZXlfbGVuID4gMCkKIAkJbWVtY3B5KHNsb3QtPnBzc19jYW5jZWxfa2V5
LCBjYW5jZWxfa2V5LCBjYW5jZWxfa2V5X2xlbik7CiAJc2xvdC0+cHNzX2NhbmNlbF9rZXlfbGVu
ID0gY2FuY2VsX2tleV9sZW47Ci0JcGdfYXRvbWljX3dyaXRlX3UzMigmc2xvdC0+cHNzX3BpZCwg
TXlQcm9jUGlkKTsKIAogCVNwaW5Mb2NrUmVsZWFzZSgmc2xvdC0+cHNzX211dGV4KTsKIAotLSAK
Mi41NC4wCgo=
--000000000000dc803506508a3cd0--