pgpool2.git / summary / log / commit / refs
commit 69419ac44d3dcde0c4bdafbf2ebf2d1137b6d273
Author: Muhammad Usama <m.usama@gmail.com>
Date: Thu Aug 29 15:34:13 2019 +0000
Add parameter enable_consensus_with_half_votes to configure majority rule calculations
Pgpool-II takes the decision of quorum existence and failover consensus after
receiving the exact 50% of votes when the watchdog cluster is configured with
an even number of nodes. With enable_consensus_with_half_votes parameter,
users can tell Pgpool-II, whether the distributed consensus in an even number
of nodes cluster requires (n/2) or ((n/2) +1) votes to decide the majority.
The patch is drafted by "Tatsuo Ishii <ishii@sraoss.co.jp>" and I have made
few modifications on top of that.
doc/src/sgml/watchdog.sgml | 60 ++++++++++++++++
src/config/pool_config_variables.c | 9 +++
src/include/pool_config.h | 7 ++
src/include/pool_config_variables.h | 2 +-
src/sample/pgpool.conf.sample | 10 +++
src/sample/pgpool.conf.sample-logical | 10 +++
src/sample/pgpool.conf.sample-master-slave | 10 +++
src/sample/pgpool.conf.sample-replication | 10 +++
src/sample/pgpool.conf.sample-stream | 10 +++
src/test/regression/tests/004.watchdog/master.conf | 2 +
.../regression/tests/004.watchdog/standby.conf | 1 +
src/watchdog/watchdog.c | 82 ++++++++++++++++++----
12 files changed, 200 insertions(+), 13 deletions(-)
diff --git a/doc/src/sgml/watchdog.sgml b/doc/src/sgml/watchdog.sgml
index 2e5ba046f..4584f510d 100644
--- a/doc/src/sgml/watchdog.sgml
+++ b/doc/src/sgml/watchdog.sgml
@@ -580,6 +580,66 @@
</listitem>
</varlistentry>
+ <varlistentry id="guc-enable-consensus-with-half-votes" xreflabel="enable_consensus_with_half_votes">
+ <term><varname>enable_consensus_with_half_votes</varname> (<type>boolean</type>)
+ <indexterm>
+ <primary><varname>cenable_consensus_with_half_votes</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ This parameter configures how the majority rule computation
+ is made by <productname>Pgpool-II</productname> for calculating
+ the quorum and resolving the consensus for failover.
+ </para>
+ <para>
+ when enabled the existence of quorum and consensus on failover requires only half of
+ the total number of votes configured in the cluster. Otherwise, both of these
+ decisions require at least one more vote than half of the total number of votes.
+ For failover, this parameter works in conjunction with the
+ <xref linkend="guc-failover-require-consensus">. In both cases, whether making a
+ decision of quorum existence or building the consensus on failover this
+ parameter only comes into play when the watchdog cluster is configured for even
+ number of <productname>Pgpool-II</productname> nodes.
+ The majority rule decision in the watchdog cluster having an odd number of participants.
+ It is not affected by the value of this configuration parameter.
+ </para>
+ <para>
+ For example, when this parameter is enabled in a two node watchdog
+ cluster, one <productname>Pgpool-II</productname> node needs to
+ be alive to make the quorum exist. If the parameter is off, two
+ nodes need to be alive to make quorum exist.
+ </para>
+ <para>
+ When this parameter is enabled in a four node watchdog cluster,
+ two <productname>Pgpool-II</productname> node needs to be alive
+ to make the quorum exist. If the parameter is off, three nodes
+ need to be alive to make quorum exist.
+ </para>
+ <para>
+ By enabling this parameter, you should aware that you take a
+ risk to make split-brain happen. For example, in four node
+ cluster consisted of node A, B, C and D, it is possible that the
+ cluster goes into two separated networks (A, B) and (C, D). For
+ (A, B) and (C, D) the quorum still exist since for both groups
+ there are two live nodes out of 4. The two groups choose their
+ own master watchdog, which is a split-brain.
+ </para>
+ <para>
+ Default is off.
+ </para>
+ <para>
+ <varname>enable_consensus_with_half_votes</varname> is not available
+ prior to <productname>Pgpool-II
+ </productname><emphasis>V4.1</emphasis>. The prior versions work
+ as if the parameter is on.
+ </para>
+ <para>
+ This parameter can only be set at server start.
+ </para>
+ </listitem>
+ </varlistentry>
+
</variablelist>
</sect2>
diff --git a/src/config/pool_config_variables.c b/src/config/pool_config_variables.c
index f291f40ed..1cdf3ed23 100644
--- a/src/config/pool_config_variables.c
+++ b/src/config/pool_config_variables.c
@@ -308,6 +308,15 @@ static struct config_bool ConfigureNamesBool[] =
false,
NULL, NULL, NULL
},
+ {
+ {"enable_consensus_with_half_votes", CFGCXT_INIT, FAILOVER_CONFIG,
+ "apply majority rule for consensus and quorum computation at 50% of votes in a cluster with an even number of nodes.",
+ CONFIG_VAR_TYPE_BOOL, false, 0
+ },
+ &g_pool_config.enable_consensus_with_half_votes,
+ false,
+ NULL, NULL, NULL
+ },
{
{"log_connections", CFGCXT_RELOAD, LOGING_CONFIG,
"Logs each successful connection.",
diff --git a/src/include/pool_config.h b/src/include/pool_config.h
index dab3fa3ed..b5dfa2c8b 100644
--- a/src/include/pool_config.h
+++ b/src/include/pool_config.h
@@ -478,6 +478,13 @@ typedef struct
* can send multiple
* failover requests to
* build consensus */
+ bool enable_consensus_with_half_votes;
+ /* apply majority rule for consensus
+ * and quorum computation at 50% of
+ * votes in a cluster with an even
+ * number of nodes.
+ */
+
WdLifeCheckMethod wd_lifecheck_method; /* method of lifecheck.
* 'heartbeat' or 'query' */
bool clear_memqcache_on_escalation; /* Clear query cache on shmem
diff --git a/src/include/pool_config_variables.h b/src/include/pool_config_variables.h
index 7bf4f51be..e3841112f 100644
--- a/src/include/pool_config_variables.h
+++ b/src/include/pool_config_variables.h
@@ -4,7 +4,7 @@
* pgpool: a language independent connection pool server for PostgreSQL
* written by Tatsuo Ishii
*
- * Copyright (c) 2003-2018 PgPool Global Development Group
+ * Copyright (c) 2003-2019 PgPool Global Development Group
*
* Permission to use, copy, modify, and distribute this software and
* its documentation for any purpose and without fee is hereby
diff --git a/src/sample/pgpool.conf.sample b/src/sample/pgpool.conf.sample
index 4d69fe510..16a650a42 100644
--- a/src/sample/pgpool.conf.sample
+++ b/src/sample/pgpool.conf.sample
@@ -646,6 +646,16 @@ allow_multiple_failover_requests_from_node = off
# for building the consensus on failover
# (change requires restart)
+enable_consensus_with_half_votes = off
+ # apply majority rule for consensus and quorum computation
+ # at 50% of votes in a cluster with even number of nodes.
+ # when enabled the existence of quorum and consensus
+ # on failover is resolved after receiving half of the
+ # total votes in the cluster, otherwise both these
+ # decisions require at least one more vote than
+ # half of the total votes.
+ # (change requires restart)
+
# - Lifecheck Setting -
# -- common --
diff --git a/src/sample/pgpool.conf.sample-logical b/src/sample/pgpool.conf.sample-logical
index 6f8cdc7c0..cf69ae765 100644
--- a/src/sample/pgpool.conf.sample-logical
+++ b/src/sample/pgpool.conf.sample-logical
@@ -626,6 +626,16 @@ enable_multiple_failover_requests_from_node = off
# A Pgpool-II node can cast multiple votes
# for building the consensus on failover
+enable_consensus_with_half_votes = off
+ # apply majority rule for consensus and quorum computation
+ # at 50% of votes in a cluster with even number of nodes.
+ # when enabled the existence of quorum and consensus
+ # on failover is resolved after receiving half of the
+ # total votes in the cluster, otherwise both these
+ # decisions require at least one more vote than
+ # half of the total votes.
+ # (change requires restart)
+
# - Lifecheck Setting -
# -- common --
diff --git a/src/sample/pgpool.conf.sample-master-slave b/src/sample/pgpool.conf.sample-master-slave
index 49bb96646..bf0ea2357 100644
--- a/src/sample/pgpool.conf.sample-master-slave
+++ b/src/sample/pgpool.conf.sample-master-slave
@@ -642,6 +642,16 @@ allow_multiple_failover_requests_from_node = off
# (change requires restart)
+enable_consensus_with_half_votes = off
+ # apply majority rule for consensus and quorum computation
+ # at 50% of votes in a cluster with even number of nodes.
+ # when enabled the existence of quorum and consensus
+ # on failover is resolved after receiving half of the
+ # total votes in the cluster, otherwise both these
+ # decisions require at least one more vote than
+ # half of the total votes.
+ # (change requires restart)
+
# - Lifecheck Setting -
# -- common --
diff --git a/src/sample/pgpool.conf.sample-replication b/src/sample/pgpool.conf.sample-replication
index b16322a79..78f9c6b5b 100644
--- a/src/sample/pgpool.conf.sample-replication
+++ b/src/sample/pgpool.conf.sample-replication
@@ -642,6 +642,16 @@ allow_multiple_failover_requests_from_node = off
# (change requires restart)
+enable_consensus_with_half_votes = off
+ # apply majority rule for consensus and quorum computation
+ # at 50% of votes in a cluster with even number of nodes.
+ # when enabled the existence of quorum and consensus
+ # on failover is resolved after receiving half of the
+ # total votes in the cluster, otherwise both these
+ # decisions require at least one more vote than
+ # half of the total votes.
+ # (change requires restart)
+
# - Lifecheck Setting -
# -- common --
diff --git a/src/sample/pgpool.conf.sample-stream b/src/sample/pgpool.conf.sample-stream
index f76d5cc10..0ab05d005 100644
--- a/src/sample/pgpool.conf.sample-stream
+++ b/src/sample/pgpool.conf.sample-stream
@@ -644,6 +644,16 @@ allow_multiple_failover_requests_from_node = off
# (change requires restart)
+enable_consensus_with_half_votes = off
+ # apply majority rule for consensus and quorum computation
+ # at 50% of votes in a cluster with even number of nodes.
+ # when enabled the existence of quorum and consensus
+ # on failover is resolved after receiving half of the
+ # total votes in the cluster, otherwise both these
+ # decisions require at least one more vote than
+ # half of the total votes.
+ # (change requires restart)
+
# - Lifecheck Setting -
# -- common --
diff --git a/src/test/regression/tests/004.watchdog/master.conf b/src/test/regression/tests/004.watchdog/master.conf
index 418f05134..f6bb6823c 100644
--- a/src/test/regression/tests/004.watchdog/master.conf
+++ b/src/test/regression/tests/004.watchdog/master.conf
@@ -12,3 +12,5 @@ heartbeat_destination_port0 = 21105
other_pgpool_hostname0 = 'localhost'
other_pgpool_port0 = 11100
other_wd_port0 = 21104
+
+enable_consensus_with_half_votes = on
diff --git a/src/test/regression/tests/004.watchdog/standby.conf b/src/test/regression/tests/004.watchdog/standby.conf
index 02353dc9a..5da16952b 100644
--- a/src/test/regression/tests/004.watchdog/standby.conf
+++ b/src/test/regression/tests/004.watchdog/standby.conf
@@ -14,3 +14,4 @@ other_pgpool_hostname0 = 'localhost'
other_pgpool_port0 = 11000
other_wd_port0 = 21004
+enable_consensus_with_half_votes = on
diff --git a/src/watchdog/watchdog.c b/src/watchdog/watchdog.c
index 403a641e2..fd6debf7f 100644
--- a/src/watchdog/watchdog.c
+++ b/src/watchdog/watchdog.c
@@ -5,7 +5,7 @@
* pgpool: a language independent connection pool server for PostgreSQL
* written by Tatsuo Ishii
*
- * Copyright (c) 2003-2016 PgPool Global Development Group
+ * Copyright (c) 2003-2019 PgPool Global Development Group
*
* Permission to use, copy, modify, and distribute this software and
* its documentation for any purpose and without fee is hereby
@@ -420,7 +420,8 @@ static void service_internal_command(void);
static unsigned int get_next_commandID(void);
static WatchdogNode * parse_node_info_message(WDPacketData * pkt, char **authkey);
static void update_quorum_status(void);
-static int get_mimimum_nodes_required_for_quorum(void);
+static int get_mimimum_remote_nodes_required_for_quorum(void);
+static int get_minimum_votes_to_resolve_consensus(void);
static bool write_packet_to_socket(int sock, WDPacketData * pkt, bool ipcPacket);
static int read_sockets(fd_set *rmask, int pending_fds_count);
@@ -2384,7 +2385,7 @@ static WDFailoverCMDResults compute_failover_consensus(POOL_REQUEST_KIND reqKind
bool duplicate = false;
WDFailoverObject *failoverObj = add_failover(reqKind, node_id_list, node_count, wdNode, *flags, &duplicate);
- if (failoverObj->request_count <= get_mimimum_nodes_required_for_quorum())
+ if (failoverObj->request_count < get_minimum_votes_to_resolve_consensus())
{
ereport(LOG, (
errmsg("failover requires the majority vote, waiting for consensus"),
@@ -6333,14 +6334,19 @@ update_quorum_status(void)
{
int quorum_status = g_cluster.quorum_status;
- if (g_cluster.clusterMasterInfo.standby_nodes_count > get_mimimum_nodes_required_for_quorum())
+ if (g_cluster.clusterMasterInfo.standby_nodes_count > get_mimimum_remote_nodes_required_for_quorum())
{
g_cluster.quorum_status = 1;
}
- else if (g_cluster.clusterMasterInfo.standby_nodes_count == get_mimimum_nodes_required_for_quorum())
+ else if (g_cluster.clusterMasterInfo.standby_nodes_count == get_mimimum_remote_nodes_required_for_quorum())
{
if (g_cluster.remoteNodeCount % 2 != 0)
- g_cluster.quorum_status = 0; /* on the edge */
+ {
+ if (pool_config->enable_consensus_with_half_votes)
+ g_cluster.quorum_status = 0; /* on the edge */
+ else
+ g_cluster.quorum_status = -1;
+ }
else
g_cluster.quorum_status = 1;
}
@@ -6355,24 +6361,76 @@ update_quorum_status(void)
}
}
-/* returns the minimum number of remote nodes required for quorum */
+/*
+ * returns the minimum number of remote nodes required for quorum
+ */
static int
-get_mimimum_nodes_required_for_quorum(void)
+get_mimimum_remote_nodes_required_for_quorum(void)
{
/*
* Even numner of remote nodes, That means total number of nodes are odd,
- * so minimum quorum is just remote/2
+ * so minimum quorum is just remote/2.
*/
if (g_cluster.remoteNodeCount % 2 == 0)
- return (g_cluster.remoteNodeCount / 2);
+ return (g_cluster.remoteNodeCount / 2);
/*
- * Total nodes including self are even, So we consider 50% nodes as
- * quorum, should we?
+ * Total nodes including self are even, So we return 50% nodes as quorum
+ * requirements
*/
return ((g_cluster.remoteNodeCount - 1) / 2);
}
+/*
+ * returns the minimum number of votes required for consensus
+ */
+static int
+get_minimum_votes_to_resolve_consensus(void)
+{
+ /*
+ * Since get_mimimum_remote_nodes_required_for_quorum() returns
+ * the number of remote nodes required to complete the quorum
+ * that is always one less than the total number of nodes required
+ * for the cluster to build quorum or consensus, reason being
+ * in get_mimimum_remote_nodes_required_for_quorum()
+ * we always consider the local node as a valid pre-casted vote.
+ * But when it comes to count the number of votes required to build
+ * consensus for any type of decision, for example for building the
+ * consensus on backend failover, the local node can vote on either
+ * side. So it's vote is not explicitly counted and for the consensus
+ * we actually need one more vote than the total number of remote nodes
+ * required for the quorum
+ *
+ * For example
+ * If Total nodes in cluster = 4
+ * remote node will be = 3
+ * get_mimimum_remote_nodes_required_for_quorum() return = 1
+ * Minimum number of votes required for consensu will be
+ *
+ * if(pool_config->enable_consensus_with_half_votes = true)
+ * (exact 50% n/2) ==> 4/2 = 2
+ *
+ * if(pool_config->enable_consensus_with_half_votes = false)
+ * (exact 50% +1 ==> (n/2)+1) ==> (4/2)+1 = 3
+ *
+ */
+
+ int required_node_count = get_mimimum_remote_nodes_required_for_quorum() + 1;
+ /*
+ * When the total number of nodes in the watchdog cluster including the
+ * local node are even, The number of votes required for the consensus
+ * depends on the enable_consensus_with_half_votes.
+ * So for even number of nodes when enable_consensus_with_half_votes is
+ * not allowed than we would nedd one more vote than exact 50%
+ */
+ if (g_cluster.remoteNodeCount % 2 != 0)
+ {
+ if (pool_config->enable_consensus_with_half_votes == false)
+ required_node_count += 1;
+ }
+
+ return required_node_count;
+}
/*
* sets the state of local watchdog node, and fires a state change event
[parent: 8b0e1ff893e0]