agora inbox for pgpool-committers@postgresql.orghelp / color / mirror / Atom feed
pgpool: Prevent watchdog split-brain scenario in some corner cases. 6+ messages / 1 participants [nested] [flat]
* pgpool: Prevent watchdog split-brain scenario in some corner cases. @ 2025-10-03 01:55 Tatsuo Ishii <ishii@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Tatsuo Ishii @ 2025-10-03 01:55 UTC (permalink / raw) To: pgpool-committers@lists.postgresql.org Prevent watchdog split-brain scenario in some corner cases. Watchdog uses two methods for communication: beacon message and heartbeat. If the heartbeat is not working yet, it is possible that split-brain occurs if beacon messages are not received by standby nodes more than 30 seconds (see watchdog_state_machine_standby). In this case other watchdog node becomes new leader node. Problem is, the former leader node remains leader too, thus there are two leader nodes: split-brain. Suppose we have 3 watchdog nodes (node 0, 1, 2) and node 2 was not started due a maintenace work. In this case life check is not started. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50000 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 7 STANDBY 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER Then to stop the beacon signal, send STOP signal to node0 watchdg process. Node 1 is promoted and now we have two leader nodes 0 and 1. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50004 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 4 LEADER 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER This commit tries to prevent having two leaders by revoking the former leader. If a node detects beacon message from leader node being lost more than 2 times (3 * BEACON_MESSAGE_INTERVAL_SECONDS), leader node is set to LOST state and new leader node election process is triggered by calling set_state(WD_JOINING). Backpatch-through: v4.2 Branch ------ V4_2_STABLE Details ------- https://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=379eb1925839568921a340571f540c7f17fea... Modified Files -------------- src/watchdog/watchdog.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgpool: Prevent watchdog split-brain scenario in some corner cases. @ 2025-10-03 01:55 Tatsuo Ishii <ishii@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Tatsuo Ishii @ 2025-10-03 01:55 UTC (permalink / raw) To: pgpool-committers@lists.postgresql.org Prevent watchdog split-brain scenario in some corner cases. Watchdog uses two methods for communication: beacon message and heartbeat. If the heartbeat is not working yet, it is possible that split-brain occurs if beacon messages are not received by standby nodes more than 30 seconds (see watchdog_state_machine_standby). In this case other watchdog node becomes new leader node. Problem is, the former leader node remains leader too, thus there are two leader nodes: split-brain. Suppose we have 3 watchdog nodes (node 0, 1, 2) and node 2 was not started due a maintenace work. In this case life check is not started. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50000 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 7 STANDBY 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER Then to stop the beacon signal, send STOP signal to node0 watchdg process. Node 1 is promoted and now we have two leader nodes 0 and 1. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50004 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 4 LEADER 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER This commit tries to prevent having two leaders by revoking the former leader. If a node detects beacon message from leader node being lost more than 2 times (3 * BEACON_MESSAGE_INTERVAL_SECONDS), leader node is set to LOST state and new leader node election process is triggered by calling set_state(WD_JOINING). Backpatch-through: v4.2 Branch ------ V4_3_STABLE Details ------- https://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=0bbd4b52d0892902798c37f8a96cdb2a79252... Modified Files -------------- src/watchdog/watchdog.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgpool: Prevent watchdog split-brain scenario in some corner cases. @ 2025-10-03 01:55 Tatsuo Ishii <ishii@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Tatsuo Ishii @ 2025-10-03 01:55 UTC (permalink / raw) To: pgpool-committers@lists.postgresql.org Prevent watchdog split-brain scenario in some corner cases. Watchdog uses two methods for communication: beacon message and heartbeat. If the heartbeat is not working yet, it is possible that split-brain occurs if beacon messages are not received by standby nodes more than 30 seconds (see watchdog_state_machine_standby). In this case other watchdog node becomes new leader node. Problem is, the former leader node remains leader too, thus there are two leader nodes: split-brain. Suppose we have 3 watchdog nodes (node 0, 1, 2) and node 2 was not started due a maintenace work. In this case life check is not started. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50000 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 7 STANDBY 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER Then to stop the beacon signal, send STOP signal to node0 watchdg process. Node 1 is promoted and now we have two leader nodes 0 and 1. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50004 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 4 LEADER 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER This commit tries to prevent having two leaders by revoking the former leader. If a node detects beacon message from leader node being lost more than 2 times (3 * BEACON_MESSAGE_INTERVAL_SECONDS), leader node is set to LOST state and new leader node election process is triggered by calling set_state(WD_JOINING). Backpatch-through: v4.2 Branch ------ V4_4_STABLE Details ------- https://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=6f0ebabe446a42786697d995644b85a0e6163... Modified Files -------------- src/watchdog/watchdog.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgpool: Prevent watchdog split-brain scenario in some corner cases. @ 2025-10-03 01:55 Tatsuo Ishii <ishii@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Tatsuo Ishii @ 2025-10-03 01:55 UTC (permalink / raw) To: pgpool-committers@lists.postgresql.org Prevent watchdog split-brain scenario in some corner cases. Watchdog uses two methods for communication: beacon message and heartbeat. If the heartbeat is not working yet, it is possible that split-brain occurs if beacon messages are not received by standby nodes more than 30 seconds (see watchdog_state_machine_standby). In this case other watchdog node becomes new leader node. Problem is, the former leader node remains leader too, thus there are two leader nodes: split-brain. Suppose we have 3 watchdog nodes (node 0, 1, 2) and node 2 was not started due a maintenace work. In this case life check is not started. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50000 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 7 STANDBY 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER Then to stop the beacon signal, send STOP signal to node0 watchdg process. Node 1 is promoted and now we have two leader nodes 0 and 1. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50004 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 4 LEADER 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER This commit tries to prevent having two leaders by revoking the former leader. If a node detects beacon message from leader node being lost more than 2 times (3 * BEACON_MESSAGE_INTERVAL_SECONDS), leader node is set to LOST state and new leader node election process is triggered by calling set_state(WD_JOINING). Backpatch-through: v4.2 Branch ------ V4_5_STABLE Details ------- https://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=c7cf990d517a5a54e9241ea1917cb5a963a2b... Modified Files -------------- src/watchdog/watchdog.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgpool: Prevent watchdog split-brain scenario in some corner cases. @ 2025-10-03 01:55 Tatsuo Ishii <ishii@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Tatsuo Ishii @ 2025-10-03 01:55 UTC (permalink / raw) To: pgpool-committers@lists.postgresql.org Prevent watchdog split-brain scenario in some corner cases. Watchdog uses two methods for communication: beacon message and heartbeat. If the heartbeat is not working yet, it is possible that split-brain occurs if beacon messages are not received by standby nodes more than 30 seconds (see watchdog_state_machine_standby). In this case other watchdog node becomes new leader node. Problem is, the former leader node remains leader too, thus there are two leader nodes: split-brain. Suppose we have 3 watchdog nodes (node 0, 1, 2) and node 2 was not started due a maintenace work. In this case life check is not started. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50000 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 7 STANDBY 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER Then to stop the beacon signal, send STOP signal to node0 watchdg process. Node 1 is promoted and now we have two leader nodes 0 and 1. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50004 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 4 LEADER 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER This commit tries to prevent having two leaders by revoking the former leader. If a node detects beacon message from leader node being lost more than 2 times (3 * BEACON_MESSAGE_INTERVAL_SECONDS), leader node is set to LOST state and new leader node election process is triggered by calling set_state(WD_JOINING). Backpatch-through: v4.2 Branch ------ V4_6_STABLE Details ------- https://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=b84553bf0834cfcf9c2718974b7a965c8253e... Modified Files -------------- src/watchdog/watchdog.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) ^ permalink raw reply [nested|flat] 6+ messages in thread
* pgpool: Prevent watchdog split-brain scenario in some corner cases. @ 2025-10-03 01:55 Tatsuo Ishii <ishii@postgresql.org> 0 siblings, 0 replies; 6+ messages in thread From: Tatsuo Ishii @ 2025-10-03 01:55 UTC (permalink / raw) To: pgpool-committers@lists.postgresql.org Prevent watchdog split-brain scenario in some corner cases. Watchdog uses two methods for communication: beacon message and heartbeat. If the heartbeat is not working yet, it is possible that split-brain occurs if beacon messages are not received by standby nodes more than 30 seconds (see watchdog_state_machine_standby). In this case other watchdog node becomes new leader node. Problem is, the former leader node remains leader too, thus there are two leader nodes: split-brain. Suppose we have 3 watchdog nodes (node 0, 1, 2) and node 2 was not started due a maintenace work. In this case life check is not started. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50000 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 7 STANDBY 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER Then to stop the beacon signal, send STOP signal to node0 watchdg process. Node 1 is promoted and now we have two leader nodes 0 and 1. $ pcp_watchdog_info -p 50005 3 3 NO localhost:50004 Linux tishii-CFSV9-2 localhost localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 4 LEADER 0 MEMBER localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER Not_Set localhost 50008 50010 0 DEAD 0 MEMBER This commit tries to prevent having two leaders by revoking the former leader. If a node detects beacon message from leader node being lost more than 2 times (3 * BEACON_MESSAGE_INTERVAL_SECONDS), leader node is set to LOST state and new leader node election process is triggered by calling set_state(WD_JOINING). Backpatch-through: v4.2 Branch ------ master Details ------- https://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=9e6408c58d50b65adb73e93c44131c1c53980... Modified Files -------------- src/watchdog/watchdog.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) ^ permalink raw reply [nested|flat] 6+ messages in thread
end of thread, other threads:[~2025-10-03 01:55 UTC | newest] Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2025-10-03 01:55 pgpool: Prevent watchdog split-brain scenario in some corner cases. Tatsuo Ishii <ishii@postgresql.org> 2025-10-03 01:55 pgpool: Prevent watchdog split-brain scenario in some corner cases. Tatsuo Ishii <ishii@postgresql.org> 2025-10-03 01:55 pgpool: Prevent watchdog split-brain scenario in some corner cases. Tatsuo Ishii <ishii@postgresql.org> 2025-10-03 01:55 pgpool: Prevent watchdog split-brain scenario in some corner cases. Tatsuo Ishii <ishii@postgresql.org> 2025-10-03 01:55 pgpool: Prevent watchdog split-brain scenario in some corner cases. Tatsuo Ishii <ishii@postgresql.org> 2025-10-03 01:55 pgpool: Prevent watchdog split-brain scenario in some corner cases. Tatsuo Ishii <ishii@postgresql.org>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox