Subject: Re: Built-in Raft replication

From: Andrey Borodin <[email-masked]>
Date: Wed, 16 Apr 2025 09:58:55 +0500

> On 16 Apr 2025, at 09:33, Ashutosh Bapat <[email-masked]> wrote:
> 
> In my experience, the load of managing hundreds of replicas which all
> participate in RAFT protocol becomes more than regular transaction
> load. So making every replica a RAFT participant will affect the
> ability to deploy hundreds of replica.

No need to make all standbys voting. And no need to make plain topology. pg_consul is using 2/3 or 3/5 HA groups, and cascades all others from HA group.
Existing tools already solve the original problem, Konstantin is just proposing to solve it in some standard “official” way.

> We may build an extension which
> has a similar role in PostgreSQL world as zookeeper in Hadoop.

Patroni, pg_consul and others already use zookeeper, etcd and similar systems for consensus.
Is it any better as extension than as etcd?

> It can
> be then used for other distributed systems as well - like shared
> nothing clusters based on FDW.

I didn’t get FDW analogy. Why other distributed systems should choose Postgres extension over Zookeeper?

> There's already a proposal to bring
> CREATE SERVER to the world of logical replication - so I see these two
> worlds uniting in future.

Again, I’m lost here. Which two worlds?

> The way I imagine it is some PostgreSQL
> instances, which have this extension installed, will act as a RAFT
> cluster (similar to Zookeeper ensemble or etcd cluster).

That’s exactly what is proposed here.

> The
> distributed system based on logical replication or FDW or both will
> use this ensemble to manage its shared state. The same ensemble can be
> shared across multiple distributed clusters if it has scaling
> capabilities.

Yes, shared DCS are common these days. AFAIK, we use one Zookeeper instance per hundred Postgres clusters to coordinate pg_consuls.

Actually, scalability is opposite to topic of this thread. Let me explain.
Currently, Postgres automatic failover tools rely on databases with built-in automatic failover. Konstantin is proposing to shorten this loop and make Postgres use its build-in automatic failover.

So, existing tooling allows you to have 3 hosts for DCS, with majority of 2 hosts able to elect new leader in case of failover.
And you can have only 2 hosts for Postgres - Primary and Standby. You can have 2 big Postgres machines with 64 CPUs. And 3 one-CPU hosts for Zookeper\etcd.

If you use build-in failover you have to resort to 3 big Postgres machines because you need 2/3 majority. Of course, you can install MySQL-stype arbiter - host that had no real PGDATA, only participates in voting. But this is a solution to problem induced by built-in autofailover.

Best regards, Andrey Borodin.

---

From: Ashutosh Bapat <[email-masked]>
Date: Wed, 16 Apr 2025 11:48:05 +0530

> Is it any better as extension than as etcd?

I feel so. An extension runs from within a postgresql process, uses
the same protocol as PostgreSQL whereas etcd is another process and
another protocol.

> I didn’t get FDW analogy. Why other distributed systems should choose Postgres extension over Zookeeper?

By other distributed systems I mean PostgreSQL distributed systems -
FDW based native sharding or native replication or a system which uses
both.

> Again, I’m lost here. Which two worlds?

Logical replication and FDW based native sharding.

> If you use build-in failover you have to resort to 3 big Postgres machines because you need 2/3 majority. Of course, you can install MySQL-stype arbiter - host that had no real PGDATA, only partici…

Users find it a waste of resources to deploy 3 big PostgreSQL
instances just for HA where 2 suffice even if they deploy 3
lightweight DCS instances. Having only some of the nodes act as DCS
and others purely PostgreSQL nodes will reduce waste of resources.

-- 
Best Wishes,
Ashutosh Bapat

---

From: Andrey Borodin <[email-masked]>
Date: Wed, 16 Apr 2025 11:27:13 +0500

> On 16 Apr 2025, at 11:18, Ashutosh Bapat <[email-masked]> wrote:
> 
> Having only some of the nodes act as DCS

But typically you need more DCS nodes than PostgreSQL nodes. Did you mean
“Having only some of nodes act as PostgreSQL and others purely DCS nodes will reduce waste of resources”?

---

From: Ashutosh Bapat <[email-masked]>
Date: Wed, 16 Apr 2025 12:15:44 +0530

> But typically you need more DCS nodes than PostgreSQL nodes. Did you mean

In a small HA setup this might be true. But not when there are many
replicas. But ...

> “Having only some nodes act as PostgreSQL and others purely DCS nodes will reduce waste of resources”?

I mean, whatever the setup may be one shouldn't require to deploy a
big PostgreSQL server just because DCS needs majority.

-- 
Best Wishes,
Ashutosh Bapat

---

From: Alastair Turner <[email-masked]>
Date: Wed, 16 Apr 2025 13:53:23 +0100

On Wed, 16 Apr 2025 at 07:18, Ashutosh Bapat <[email-masked]>
wrote:

> > If you use build-in failover you have to resort to 3 big Postgres
> machines because you need 2/3 majority. Of course, you can install
> MySQL-stype arbiter - host that had no real PGDATA, only participates in
> voting. But this is a solution to problem induced by built-in autofailover.
>
> The experience of other projects/products with automated failover based on
quorum shows that this is a critical issue for adoption. In the In-memory
Data Grid space (Coherence, Geode/GemFire) the question of how to ensure
that some nodes didn't carry any data comes up early in many architecture
discussions. When RabbitMQ shipped their Quorum Queues feature, the first
and hardest area of pushback was around all nodes hosting message content.

It's not just about the requirement for compute resources, it's also about
bandwidth and latency. Many large organisations have, for historical
reasons, pairs of data centres with very good point-to-point connectivity.
As the requirement for quorum witnesses has come up for all sorts of
things, including storage arrays, they have built arbiter/witness sites at
branches, colocation providers or even on the public cloud. More than not
holding user data or processing queries, the arbiter can't even be sent the
replication stream for the user data in the database, it just won't fit
down the pipe.

Which feels like a very difficult requirement to meet if the replication
model for all data is being changed to a quorum model.

Regards
Alastair

---

From: Konstantin Osipov <[email-masked]>
Date: Wed, 16 Apr 2025 17:07:36 +0300

> model for all data is being changed to a quorum model.

I agree master/replica deployment layouts are very popular and are
not going to directly benefit from raft. They'll still work, but no
automation will be available, just like today with Patroni.

However, if the storage cost is an argument, then the logical path is to
disaggregate storage/compute altogether, i.e. use projects like
neon.

-- 
Konstantin Osipov, Moscow, Russia

---

From: Greg Sabino Mullane <[email-masked]>
Date: Wed, 16 Apr 2025 15:29:05 -0400

On Wed, Apr 16, 2025 at 2:18 AM Ashutosh Bapat <[email-masked]>
wrote:

> Users find it a waste of resources to deploy 3 big PostgreSQL instances
> just for HA where 2 suffice even if they deploy 3 lightweight DCS
> instances. Having only some of the nodes act as DCS and others purely
> PostgreSQL nodes will reduce waste of resources.
>

A big problem is that putting your DCS into Postgres means your whole
system is now super-sensitive to IO/WAL-streaming issues, and a busy
database doing database stuff is going to start affecting the DCS stuff.
With three lightweight DCS servers, you don't really need to worry about
how stressed the database servers are. In that way, I feel etcd et al. are
adhering to the unix philosophy of "do one thing, and do it well."

Cheers,
Greg

--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support

---

From: Konstantin Osipov <[email-masked]>
Date: Wed, 16 Apr 2025 22:35:34 +0300

> database doing database stuff is going to start affecting the DCS stuff.

Affecting in what way? Do you have a scenario in mind where an
external state provider would act differently (better)?

---

From: Alastair Turner <[email-masked]>
Date: Wed, 16 Apr 2025 22:45:45 +0100

Hi Konstantin

On Wed, 16 Apr 2025 at 15:07, Konstantin Osipov <[email-masked]>
wrote:

> Users of Patroni and etcd setups can get automation for two-site
primary/replica pairs by putting a third etcd node on a third site. Which
only requires moving the membership/leadership data to the arbiter site,
not all database activity.

> The issue is not generally storage, but network. There may simply not be
enough bandwidth available to transmit the whole WAL to the arbiter site.

Many on-premises IT setups have this limitation in some form.

If your proposal would leave these large, traditional user organisations
(which account for thousands of Postgres HA pairs or DR pairs) doing what
they currently do with wraparound tooling like Patroni, and create a new,
in core, option for balanced 3, 5, 7... member groups, then I don't think
it's worth doing.

Regards,
Alastair