public inbox for [email protected]  

help / color / mirror / Atom feed

From: Tatsuo Ishii <[email protected]>
To: [email protected]
Subject: pgpool: Prevent watchdog split-brain scenario in some corner cases.
Date: Fri, 03 Oct 2025 01:55:29 +0000
Message-ID: <[email protected]> (raw)

Prevent watchdog split-brain scenario in some corner cases.

Watchdog uses two methods for communication: beacon message and
heartbeat.  If the heartbeat is not working yet, it is possible that
split-brain occurs if beacon messages are not received by standby
nodes more than 30 seconds (see watchdog_state_machine_standby). In
this case other watchdog node becomes new leader node. Problem is, the
former leader node remains leader too, thus there are two leader
nodes: split-brain.

Suppose we have 3 watchdog nodes (node 0, 1, 2) and node 2 was not
started due a maintenace work. In this case life check is not started.

$ pcp_watchdog_info -p 50005
3 3 NO localhost:50000 Linux tishii-CFSV9-2 localhost

localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 7 STANDBY 0 MEMBER
localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER
Not_Set localhost 50008 50010 0 DEAD 0 MEMBER

Then to stop the beacon signal, send STOP signal to node0 watchdg
process. Node 1 is promoted and now we have two leader nodes 0 and 1.

$ pcp_watchdog_info -p 50005
3 3 NO localhost:50004 Linux tishii-CFSV9-2 localhost

localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 4 LEADER 0 MEMBER
localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER
Not_Set localhost 50008 50010 0 DEAD 0 MEMBER

This commit tries to prevent having two leaders by revoking the former
leader. If a node detects beacon message from leader node being lost
more than 2 times (3 * BEACON_MESSAGE_INTERVAL_SECONDS), leader node
is set to LOST state and new leader node election process is triggered
by calling set_state(WD_JOINING).

Backpatch-through: v4.2

Branch
------
V4_6_STABLE

Details
-------
https://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=b84553bf0834cfcf9c2718974b7a965c8253e...

Modified Files
--------------
src/watchdog/watchdog.c | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected]
  Subject: Re: pgpool: Prevent watchdog split-brain scenario in some corner cases.
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox