Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wfC1L-005K9x-15 for pgpool-hackers@arkaria.postgresql.org; Thu, 02 Jul 2026 07:39:39 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1wfC1J-000qbN-0Y for pgpool-hackers@arkaria.postgresql.org; Thu, 02 Jul 2026 07:39:37 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wfC1I-000qbF-2M for pgpool-hackers@lists.postgresql.org; Thu, 02 Jul 2026 07:39:37 +0000 Received: from meldrar.postgresql.org ([2a02:c0:301:0:ffff::31]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1wfC1G-00000001LHg-1On5 for pgpool-hackers@lists.postgresql.org; Thu, 02 Jul 2026 07:39:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=postgresql.org; s=20171124; h=Content-Transfer-Encoding:Content-Type: Mime-Version:References:In-Reply-To:From:Subject:Cc:To:Message-Id:Date:Sender :Reply-To:Content-ID:Content-Description; bh=sxG06bLTiAgPIB0Qu8YRFO+tGlh4XycQ0E1re4IA36s=; b=v6cDPJZdTgZ0siSRRZEDrHtvFE tM9JmzhA/LoU3uhszxLqtdWyTgrp1fTVD0rYOS8byKG0Ai59DuFqPF/eM6zrmaG7VKmesAnQvprfn l2ypTn3DkRpZQuacf7l6DV4ATkqfSk1VdXKws0Oq+F4oJOsuEDONgOWcwFR4zM3NA8m4BHefbhvok l5aI65oqcnUcOHO4RSXSAvpbwrQlZC/qBWeUAWzZQhRc1RMea44dtuvCU77XoP9OTldmos80BrYs8 6AF0FJbRNod4DymM4jGYrMR0VQk0O7SaQ5S7sWxw+9oikGgd9qQ7QcpbujEHXKgygrOnLrKm+HrjH fs89GeaQ==; Received: from [2409:11:4120:300:67d8:5815:403d:7e85] (helo=localhost) by meldrar.postgresql.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1wfC1E-008yA0-06; Thu, 02 Jul 2026 07:39:34 +0000 Date: Thu, 02 Jul 2026 16:39:22 +0900 (JST) Message-Id: <20260702.163922.30147080599128209.ishii@postgresql.org> To: emond.papegaaij@gmail.com Cc: pgpool-hackers@lists.postgresql.org Subject: Re: [PATCH] Segfault in pool_do_auth() when failover races with a new connection From: Tatsuo Ishii In-Reply-To: References: X-Mailer: Mew version 6.8 on Emacs 29.3 Mime-Version: 1.0 Content-Type: Multipart/Mixed; boundary="--Next_Part(Thu_Jul__2_16_39_22_2026_100)--" Content-Transfer-Encoding: 7bit X-Host-Lookup-Failed: Reverse DNS lookup failed for 2409:11:4120:300:67d8:5815:403d:7e85 (failed) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk ----Next_Part(Thu_Jul__2_16_39_22_2026_100)-- Content-Type: Text/Plain; charset=utf-8 Content-Transfer-Encoding: base64 SGkgRW1vbmQsDQoNCj4gSGksDQo+IA0KPiBMYXN0IG5pZ2h0IG9uZSBvZiBDSSBidWlsZHMgZGV0 ZWN0ZWQgYSBzZWdtZW50YXRpb24gZmF1bHQgaW4gcGdwb29sDQo+ICg0LjcuMikuIEkndmUgcmFu IGFuIGFuYWx5c2lzIG9uIHRoZSBjb3JlZHVtcCB3aXRoIENsYXVkZSBhbmQgY2FtZSB0bw0KPiB0 aGUgY29uY2x1c2lvbiB0aGF0IHRoaXMgY2FuIGhhcHBlbiB3aGVuIHRyeWluZyB0byBvcGVuIGEg Y29ubmVjdGlvbg0KPiBpbiB0aGUgbWlkZGxlIG9mIGEgZmFpbG92ZXIuDQo+IA0KPiBTeW1wdG9t IC8gYmFja3RyYWNlIChyZXNvbHZlZCBmcm9tIHRoZSBjb3JlIGR1bXApOg0KPiBwb29sX2RvX2F1 dGggKHNyYy9hdXRoL3Bvb2xfYXV0aC5jOjM0OSkgICA8LS0gU0lHU0VHViwgZmF1bHQgYWRkciAw eDANCj4gY29ubmVjdF9iYWNrZW5kIChzcmMvcHJvdG9jb2wvY2hpbGQuYzoxMDk5KSAgKFBHX1RS WSBibG9jaykNCj4gZ2V0X2JhY2tlbmRfY29ubmVjdGlvbiAoc3JjL3Byb3RvY29sL2NoaWxkLmM6 MjEwOSkNCj4gZG9fY2hpbGQgKHNyYy9wcm90b2NvbC9jaGlsZC5jOjIzMykNCj4gDQo+IFRoZSBj cmFzaGluZyBzdGF0ZW1lbnQgaXM6DQo+IHByb3RvTWFqb3IgPSBNQUlOX0NPTk5FQ1RJT04oY3Ap LT5zcC0+bWFqb3I7ICAgLyogcG9vbF9hdXRoLmM6MzQ5ICovDQo+IGkuZS4gY3AtPnNsb3RzW01B SU5fTk9ERV9JRF0gaXMgTlVMTC4gRnJvbSB0aGUgY29yZSwgdGhlIHBvb2wgaGFkDQo+IHNsb3Rz WzBdID09IE5VTEwgd2hpbGUgc2xvdHNbMV0vc2xvdHNbMl0gd2VyZSB2YWxpZCwgYW5kIE1BSU5f Tk9ERV9JRCA9PSAwLg0KPiANCj4gcG9vbF9pbml0aWFsaXplX3ByaXZhdGVfYmFja2VuZF9zdGF0 dXMoKSBzbmFwc2hvdHMNCj4gcHJpdmF0ZV9iYWNrZW5kX3N0YXR1c1tdIGFuZCBteV9tYWluX25v ZGVfaWQgbm9u4oCRYXRvbWljYWxseSB3LnIudC4gdGhlDQo+IHBhcmVudCdzIGZhaWxvdmVyLCBz byBhIGNoaWxkIGNhbiBjYXB0dXJlIG15X21haW5fbm9kZV9pZCA9PSAwIHdoaWxlDQo+IG5vZGUg MCBpcyBhbHJlYWR5IENPTl9ET1dOLiBuZXdfY29ubmVjdGlvbigpIHRoZW4gbmV2ZXIgY3JlYXRl cw0KPiBzbG90c1swXSwgYW5kIHBvb2xfZG9fYXV0aCgpIGRlcmVmZXJlbmNlcyB0aGUgTlVMTCBt YWluIHNsb3QuIChTYW1lDQo+IGNsYXNzIGFzIHRoZSBOVUxM4oCRTUFJTl9DT05ORUNUSU9OIGNh c2Ugbm90ZWQgaW4NCj4gcG9vbF92aXJ0dWFsX21haW5fZGJfbm9kZV9pZCgpLikNCg0KRnJvbSBw b29sX2luaXRpYWxpemVfcHJpdmF0ZV9iYWNrZW5kX3N0YXR1cygpOg0KDQoJZm9yIChpID0gMDsg aSA8IE1BWF9OVU1fQkFDS0VORFM7IGkrKykNCgl7DQoJCXByaXZhdGVfYmFja2VuZF9zdGF0dXNb aV0gPSBCQUNLRU5EX0lORk8oaSkuYmFja2VuZF9zdGF0dXM7DQoJCS8qIG15X2JhY2tlbmRfc3Rh dHVzIGlzIHJlZmVycmVkIHRvIGJ5IFZBTElEX0JBQ0tFTkQgbWFjcm8uICovDQoJCW15X2JhY2tl bmRfc3RhdHVzW2ldID0gJnByaXZhdGVfYmFja2VuZF9zdGF0dXNbaV07DQoJfQ0KDQoJbXlfbWFp bl9ub2RlX2lkID0gUkVBTF9NQUlOX05PREVfSUQ7DQoNCkJBQ0tFTkRfSU5GTyhpKS5iYWNrZW5k X3N0YXR1cyBhcmUgdXBkYXRlZCBpbiB0aGUgcGdwb29sIG1haW4gcHJvY2Vzcw0KYXQgbGluZSAx NzkzIG9mIGZhaWxvdmVyKHBncG9vbF9tYWluLmMpIGJ5IGNhbGxpbmcNCmhhbmRsZV9mYWlsb3Zl cl9yZXF1ZXN0KCksIGJ1dCBSRUFMX01BSU5fTk9ERV9JRCAoYWN0dWFsbHkNClJlcV9pbmZvLT5t YWluX25vZGVfaWQsIHNoYXJlZCBtZW1vcnkpIGlzIHVwZGF0ZWQgYXQgMTg0NiBvZg0KZmFpbG92 ZXIoKSBieSBjYWxsaW5nIHNhdmVfbm9kZV9pbmZvKCkuIFNvIHVwZGF0aW5nIFJFQUxfTUFJTl9O T0RFX0lEDQp0ZW5kcyB0byBzbGlnaHRseSBkZWxheSBhZnRlciB1cGRhdGluZw0KQkFDS0VORF9J TkZPKGkpLmJhY2tlbmRfc3RhdHVzLiBUaGlzIG1heSBleHBsYWluIHdoeSBteV9tYWluX25vZGUg aXMNCnNldCB0aGUgc3RhbGUgZGF0YSAoMSkuDQoNCkFub3RoZXIgcG9zc2liaWxpdHkgaXMgbXlf bWFpbl9ub2RlX2lkIGJlaW5nIGxhY2tpbmcgdm9sYXRpbGUNCnF1YWxpZmljYXRpb24uIFRoaXMg YWxzbyBjb3VsZCBzZXQgc3RhbGUgZGF0YSB0byBteV9tYWluX25vZGVfaWQgKDIpLg0KDQpPZiBj b3Vyc2UgKDEpIGFuZCAoMikgYXJlIG5vdCBtdXR1YWxseSBleGNsdXNpdmUuIFNvIHlvdXIgcGF0 Y2ggZm9yDQooMSkgYW5kIGZpeCBmb3IgKDIpIG5lZWQgdG8gYmUgYm90aCBhcHBsaWVkLg0KDQpC eSB0aGUgd2F5IGluIHlvdXIgcGF0Y2ggdGhlcmUgaXMgbG9vcHMgbGlrZSB0aGlzLiBUaGlzIGNh biBiZQ0KcmVwbGFjZWQgYnkgZXhpc3RpbmcgZ2V0X25leHRfbWFpbl9ub2RlKCkuDQoNCisJCWZv ciAoaSA9IDA7IGkgPCBOVU1fQkFDS0VORFM7IGkrKykNCisJCXsNCisJCQlpZiAocHJpdmF0ZV9i YWNrZW5kX3N0YXR1c1tpXSA9PSBDT05fVVAgfHwNCisJCQkJcHJpdmF0ZV9iYWNrZW5kX3N0YXR1 c1tpXSA9PSBDT05fQ09OTkVDVF9XQUlUKQ0KKwkJCXsNCisJCQkJbXlfbWFpbl9ub2RlX2lkID0g aTsNCisJCQkJYnJlYWs7DQorCQkJfQ0KKwkJfQ0KDQpBbGwgb2YgdGhlIGFib3ZlIGluIG1pbmQs IEkgcHJvcG9zZSBhdHRhY2hlZCBwYXRjaC4NCg0KV2hhdCBkbyB5b3UgdGhpbms/DQotLQ0KVGF0 c3VvIElzaGlpDQpTUkEgT1NTIEsuSy4NCkVuZ2xpc2g6IGh0dHA6Ly93d3cuc3Jhb3NzLmNvLmpw L2luZGV4X2VuLw0KSmFwYW5lc2U6aHR0cDovL3d3dy5zcmFvc3MuY28uanANCg== ----Next_Part(Thu_Jul__2_16_39_22_2026_100)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="connect_crash_fix_v2.patch" diff --git a/src/include/pool.h b/src/include/pool.h index 549aed30f..7ead2e926 100644 --- a/src/include/pool.h +++ b/src/include/pool.h @@ -345,8 +345,8 @@ extern int pool_get_major_version(void); extern bool pool_is_node_to_be_sent_in_current_query(int node_id); extern int pool_virtual_main_db_node_id(void); -extern BACKEND_STATUS *my_backend_status[]; -extern int my_main_node_id; +extern volatile BACKEND_STATUS *my_backend_status[]; +extern volatile int my_main_node_id; #define VALID_BACKEND(backend_id) \ ((RAW_MODE && (backend_id) == REAL_MAIN_NODE_ID) || \ @@ -662,7 +662,7 @@ extern volatile sig_atomic_t ignore_sigusr1; #define QUERY_STRING_BUFFER_LEN 1024 extern char query_string_buffer[]; /* last query string sent to simpleQuery() */ -extern BACKEND_STATUS private_backend_status[MAX_NUM_BACKENDS]; +extern volatile BACKEND_STATUS private_backend_status[MAX_NUM_BACKENDS]; extern char remote_host[]; /* client host */ extern char remote_port[]; /* client port */ diff --git a/src/main/pgpool_main.c b/src/main/pgpool_main.c index 19d89cb79..a99b1b5b5 100644 --- a/src/main/pgpool_main.c +++ b/src/main/pgpool_main.c @@ -226,7 +226,7 @@ static pid_t health_check_pids[MAX_NUM_BACKENDS]; /* * Private copy of backend status */ -BACKEND_STATUS private_backend_status[MAX_NUM_BACKENDS]; +volatile BACKEND_STATUS private_backend_status[MAX_NUM_BACKENDS]; /* * shmem connection info table @@ -278,7 +278,7 @@ static pid_t wd_lifecheck_pid = 0; /* pid for child process handling watchdog * lifecheck */ BACKEND_STATUS *my_backend_status[MAX_NUM_BACKENDS]; /* Backend status buffer */ -int my_main_node_id; /* Main node id buffer */ +volatile int my_main_node_id; /* Main node id buffer */ /* * Dummy variable to suppress compiler warnings by discarding return values diff --git a/src/protocol/child.c b/src/protocol/child.c index 4a527c84c..67a3c6c9a 100644 --- a/src/protocol/child.c +++ b/src/protocol/child.c @@ -1483,7 +1483,20 @@ pool_initialize_private_backend_status(void) my_backend_status[i] = &private_backend_status[i]; } - my_main_node_id = REAL_MAIN_NODE_ID; + my_main_node_id = REAL_MAIN_NODE_ID; + + /* + * REAL_MAIN_NODE_ID and the per-node status are read from shared memory + * non-atomically, so a concurrent failover can leave my_main_node_id + * pointing at a node we just captured as down. Re-point it at the first + * node that is up in our private snapshot; otherwise MAIN_CONNECTION() + * would dereference a connection slot that is never created for a down + * node. + */ + if (my_main_node_id < 0 || my_main_node_id >= NUM_BACKENDS || + (private_backend_status[my_main_node_id] != CON_UP && + private_backend_status[my_main_node_id] != CON_CONNECT_WAIT)) + my_main_node_id = get_next_main_node(); } static void diff --git a/src/protocol/pool_connection_pool.c b/src/protocol/pool_connection_pool.c index b16ccc39e..a939b08a7 100644 --- a/src/protocol/pool_connection_pool.c +++ b/src/protocol/pool_connection_pool.c @@ -1035,6 +1035,26 @@ new_connection(POOL_CONNECTION_POOL *p) if (active_backend_count > 0) { + /* + * A concurrent failover may have changed backend status while we were + * creating connections, leaving my_main_node_id pointing at a node we + * skipped (its slot is NULL). Re-point it at a node we did connect to + * so that MAIN_CONNECTION() never dereferences a NULL slot (e.g. in + * pool_do_auth()). + */ + if (my_main_node_id < 0 || my_main_node_id >= NUM_BACKENDS || + p->slots[my_main_node_id] == NULL) + { + for (i = 0; i < NUM_BACKENDS; i++) + { + if (p->slots[i] != NULL) + { + my_main_node_id = i; + break; + } + } + } + return p; } ----Next_Part(Thu_Jul__2_16_39_22_2026_100)----