From: Kevin Grittner <kgrittn@ymail.com>
To: HEMPLEMAN Matthew <matthew.hempleman@alstom.com>
To: pgsql-admin@postgresql.org <pgsql-admin@postgresql.org>
Subject: Re: Replication Cluster Monitoring
Date: Fri, 7 Aug 2015 16:12:18 +0000 (UTC)
Message-ID: <638490037.514417.1438963938801.JavaMail.yahoo@mail.yahoo.com> (raw)
In-Reply-To: <AE4EB74F2AE1C34FB54435AD9878F63652743C@041-DB3MPN1-027.041d.mgd.msft.net>
References: <AE4EB74F2AE1C34FB54435AD9878F63652743C@041-DB3MPN1-027.041d.mgd.msft.net>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-admin>
HEMPLEMAN Matthew <matthew.hempleman@alstom.com> wrote:
> I’m writing a Java application to monitor a streaming
> replication cluster (Windows). I want to monitor the Master and
> initiate failover if necessary (something like a scaled down
> version of pgpool). I also want to monitor the standby and
> terminate synchronous replication in the event of a failure. At
> this point, my app is polling the Master every N seconds and
> triggering a failover if the wait is too long or it receives a
> connection error. I’m worried that this method of assessing
> server health could lead to false-failovers. Any suggestions as
> to specific health checks I could run or issues I should watch
> out for?
Such an approach has many race conditions that can cause problems.
You may want to do web searches on the terms "split-brain
syndrome", STONITH, fencing, and heartbeat (as they apply to
computing). It is not trivial to get this right, and if it's not
right it can easily cause more down time than it prevents. (That's
not unique to PostgreSQL; it's the nature of automating fail-over.)
Be sure to consider what happens for transient network failures on
each machine and combination of machines, or if a machine
temporarily has a load that causes it not to respond for seconds or
minutes.
--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: kgrittn@ymail.com, matthew.hempleman@alstom.com
Subject: Re: Replication Cluster Monitoring
In-Reply-To: <638490037.514417.1438963938801.JavaMail.yahoo@mail.yahoo.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox