Subject: Built-in Raft replication

From: Konstantin Osipov <[email-masked]>
Date: Mon, 14 Apr 2025 20:15:23 +0300

Hi,

I am considering starting work on implementing a built-in Raft
replication for PostgreSQL.

Raft's advantage is that it unifies log replication, cluster
configuration/membership/topology management and initial state transfer 
into a single protocol. 

Currently the cluster configuration/topology is often managed by
Patroni, or similar tools, however, it seems there are certain
usability drawbacks with this approach: 

- it's a separate tool, requiring an external state provider like etcd;
  raft could store its configuration in system tables; this is
  also an observability improvement since everyone could look up 
  cluster state the same way as everything else

- same for watchdog; raft has a built-in failure detector that's
  configuration aware;

- flexible quorums; currently quorum size is a configurable; 
  with a built-in raft, extending the quorum could be a matter
  of starting a new node and pointing it to an existing cluster

Going forward I can see PostgreSQL providing transparent bouncing
on pg_wire level, given that Raft state is now part of the
system, so drivers and all cluster nodes could easily see where
the leader is. 

If anyone is working on Raft already I'd be happy to discuss
the details. I am fairly new to the PostgreSQL hackers ecosystem
so cautious of starting work in isolation/knowing there is no
interest in accepting the feature into the trunk.

Thanks,

-- 
Konstantin Osipov

---

From: Kirill Reshke <[email-masked]>
Date: Mon, 14 Apr 2025 22:44:32 +0500

> Hi,

Hi

> replication for PostgreSQL.

Just some thought on top of my mind, if you need my voice here:

I have a hard time believing the community will be positive about this
change in-core. It has more changes as contrib extension. In fact, if
we want a built-in consensus algorithm, Paxos is a better option,
because you can use postgresql as local crash-safe storage for single
decree paxos, just store your state (ballot number, last voice) in a
heap table.
OTOH Raft needs to write its own log, and what's worse, it sometimes
needs to remove already written parts of it (so, it is not appended
only, unlike WAL). If you have a production system which maintains two
kinds of logs with different semantics, it is a very hard system to
maintain..

There is actually a prod-ready (non open source) implementation of
RAFT as extension, called BiHA, by pgpro.

Just some thought on top of my mind, if you need my voice here.

-- 
Best regards,
Kirill Reshke

---

From: Konstantin Osipov <[email-masked]>
Date: Mon, 14 Apr 2025 22:00:22 +0300

> decree paxos, just store your state (ballot number, last voice) in a

But Raft is a log replication algorithm, not a consensus
algorithm. It does use consensus, but that's for leader election.
Paxos could be used for log replication, but that would be
expensive. In fact etcd uses Raft, and etcd is used by Patroni. So
I completely lost your line of thought here.

> kinds of logs with different semantics, it is a very hard system to

My proposal is exactly to replace (or rather, extend) the current
synchronous log replication with Raft. Entry removal is possible to
stack on top of append-only format, and production implementations
exist which do that.

So, no, it's a single log, and in fact the current WAL will do.

> RAFT as extension, called BiHA, by pgpro.

My guess biha is an extension since a proprietary code is easier
to maintain that way. I'd rather say the fact that there is a
proprietary implementation out in the field confirms it could be a
good idea to have it in PostgreSQL trunk. 

In any case I'm interested in contributing to the trunk, not
building a proprietary module/fork.

-- 
Konstantin Osipov

---

From: Aleksander Alekseev <[email-masked]>
Date: Tue, 15 Apr 2025 13:20:11 +0300

Hi Konstantin,

> replication for PostgreSQL.

Generally speaking I like the idea. The more important question IMO is
whether we want to maintain Raft within the PostgreSQL core project.

Building distributed systems on commodity hardware was a popular idea
back in the 2000s. These days you can rent a server with 2 Tb of RAM
for something like 2000 USD/month (numbers from my memory that were
valid ~5 years ago) which will fit many of the existing businesses (!)
in memory. And you can rent another one for a replica, just in order
not to recover from a backup if something happens to your primary
server. The common wisdom is if you can avoid building distributed
systems, don't build one.

Which brings the question if we want to maintain something like this
(which will include logic for cases when a node joins or leaves the
cluster, proxy server / service discovery for clients, test cases /
infrastructure for all this and also upgrading the cluster, docs, ...)
for a presumably view users which business doesn't fit in a single
server *and* they want an automatic failover (not the manual one)
*and* they don't use Patroni/Stolon/CockroachDB/Neon/... already.

Although the idea is tempting personally I'm inclined to think that
it's better to invest community resources into something else.

-- 
Best regards,
Aleksander Alekseev

---

From: Konstantin Osipov <[email-masked]>
Date: Tue, 15 Apr 2025 14:14:35 +0300

* Yura Sokolov <[email-masked]> [25/04/15 12:02]:

> Raft is log replication protocol which uses log position and term.
> But... PostgreSQL already have log position and term in its WAL structure.
> PostgreSQL's timeline is actually the Term.
> Raft implementer needs just to correct rules for Term/Timeline switching:
> - instead of "next TimeLine number is just increment of largest known
> TimeLine number" it needs to be "next TimeLine number is the result of
> Leader Election".
> 
> And yes, "it sometimes needs to remove already written parts of it".
> But... It is exactly what every PostgreSQL's cluster manager software have
> to do to join previous leader as a follower to new leader - pg_rewind.
> 
> So, PostgreSQL already have 70-90%% of Raft implementation details.
> Raft doesn't have to be implemented in PostgreSQL.
> Raft has to be finished!!!
> 
> PS: One of the biggest issues is forced snapshot on replica promotion. It
> really slows down leader switch time. It looks like it is not really
> needed, or some small workaround should be enough.

I'd say my pet peeve is storing the cluster topology (the so
called raft configuration) inside the database, not in an external
state provider. Agree on other points.

-- 
Konstantin Osipov

---

From: Aleksander Alekseev <[email-masked]>
Date: Tue, 15 Apr 2025 14:15:36 +0300

Hi Yura,

> I've been working in a company which uses MongoDB (3.6 and up) as their
> primary storage. And it seemed to me as "God Send". Everything just worked.
> Replication was as reliable as one could imagine. It outlives several
> hardware incidents without manual intervention. It allowed cluster
> maintenance (software and hardware upgrades) without application downtime.
> I really dream PostgreSQL will be as reliable as MongoDB without need of
> external services.

I completely understand. I had exactly the same experience with
Stolon. Everything just worked. And the setup took like 5 minutes.

It's a pity this project doesn't seem to get as much attention as
Patroni. Probably because attention requires traveling and presenting
the project at conferences which costs money. Or perhaps people are
just happy with Patroni. I'm not sure in which state Stolon is today.

-- 
Best regards,
Aleksander Alekseev

---

From: Yura Sokolov <[email-masked]>
Date: Tue, 15 Apr 2025 14:20:24 +0300

> just happy with Patroni. I'm not sure in which state Stolon is today.

But the key point: if PostgreSQL will be improved a bit, there will be no
need neither in Patroni, nor in Stolon. Isn't it great?

-- 
regards
Yura Sokolov aka funny-falcon

---

From: Konstantin Osipov <[email-masked]>
Date: Tue, 15 Apr 2025 14:23:14 +0300

> it's better to invest community resources into something else.

My personal take away from this as a community member would be
seamless coordinator failover in Greenplum and all of its forks
(CloudBerry, Greengage, synxdata, what not). I also imagine there
is a number of PostgreSQL derivatives that could benefit from
built-in transparent failover since it standardizes the solution
space.

-- 
Konstantin Osipov

---

From: Konstantin Osipov <[email-masked]>
Date: Tue, 15 Apr 2025 14:24:33 +0300

> I really dream PostgreSQL will be as reliable as MongoDB without need of

thanks for pointing out mongodb, so built-in raft would help
ferretdb as well.

-- 
Konstantin Osipov

---

From: Greg Sabino Mullane <[email-masked]>
Date: Tue, 15 Apr 2025 11:07:42 -0400

> interest in accepting the feature into the trunk.

Putting aside the technical concerns about this specific idea, it's best to
start by laying out a very detailed plan of what you would want to change,
and what you see as the costs and benefits. It's also extremely helpful to
think about developing this as an extension. If you get stuck due to
extension limitations, propose additional hooks. If the hooks will not
work, explain why.

Getting this into core is going to be a long, multi-year effort, in which
people are going to be pushing back the entire time, so prepare yourself
for that. My immediate retort is going to be: why would we add this if
there are existing tools that already do the job just fine? Postgres has
lots of tasks that it is happy to let other programs/OS
subsystems/extensions/etc. handle instead.

Cheers,
Greg

--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support

---

From: Konstantin Osipov <[email-masked]>
Date: Tue, 15 Apr 2025 20:27:34 +0300

> subsystems/extensions/etc. handle instead.

I had hoped I explained why external state providers can not
provide the same seamless UX as built-in ones. The key idea is to
have a built-in configuration management, so that adding and
removing replicas does not require changes in multiple disjoint
parts of the installation (server configurations, proxies,
clients).

I understand and accept that it's a multi-year effort, but I do
not accept the retort - my main point is that external tools
are not a replacement, and I'd like to reach consensus on that.

-- 
Konstantin Osipov, Moscow, Russia

---

From: Nikolay Samokhvalov <[email-masked]>
Date: Tue, 15 Apr 2025 15:27:56 -0700

On Tue, Apr 15, 2025 at 8:08 AM Greg Sabino Mullane <[email-masked]>
wrote:

> Putting aside the technical concerns about this specific idea, it's best
> to start by laying out a very detailed plan of what you would want to
> change, and what you see as the costs and benefits. It's also extremely
> helpful to think about developing this as an extension. If you get stuck
> due to extension limitations, propose additional hooks. If the hooks will
> not work, explain why.
>

This is exactly what I wanted to write as well. The idea is great. At the
same time, I think, consensus on many decisions will be extremely hard to
reach, so this project has a high risk of being very long. Unless it's an
extension, at least in the beginning.

Nik

---

From: Andrey Borodin <[email-masked]>
Date: Wed, 16 Apr 2025 09:58:55 +0500

> On 16 Apr 2025, at 09:33, Ashutosh Bapat <[email-masked]> wrote:
> 
> In my experience, the load of managing hundreds of replicas which all
> participate in RAFT protocol becomes more than regular transaction
> load. So making every replica a RAFT participant will affect the
> ability to deploy hundreds of replica.

No need to make all standbys voting. And no need to make plain topology. pg_consul is using 2/3 or 3/5 HA groups, and cascades all others from HA group.
Existing tools already solve the original problem, Konstantin is just proposing to solve it in some standard “official” way.

> We may build an extension which
> has a similar role in PostgreSQL world as zookeeper in Hadoop.

Patroni, pg_consul and others already use zookeeper, etcd and similar systems for consensus.
Is it any better as extension than as etcd?

> It can
> be then used for other distributed systems as well - like shared
> nothing clusters based on FDW.

I didn’t get FDW analogy. Why other distributed systems should choose Postgres extension over Zookeeper?

> There's already a proposal to bring
> CREATE SERVER to the world of logical replication - so I see these two
> worlds uniting in future.

Again, I’m lost here. Which two worlds?

> The way I imagine it is some PostgreSQL
> instances, which have this extension installed, will act as a RAFT
> cluster (similar to Zookeeper ensemble or etcd cluster).

That’s exactly what is proposed here.

> The
> distributed system based on logical replication or FDW or both will
> use this ensemble to manage its shared state. The same ensemble can be
> shared across multiple distributed clusters if it has scaling
> capabilities.

Yes, shared DCS are common these days. AFAIK, we use one Zookeeper instance per hundred Postgres clusters to coordinate pg_consuls.

Actually, scalability is opposite to topic of this thread. Let me explain.
Currently, Postgres automatic failover tools rely on databases with built-in automatic failover. Konstantin is proposing to shorten this loop and make Postgres use its build-in automatic failover.

So, existing tooling allows you to have 3 hosts for DCS, with majority of 2 hosts able to elect new leader in case of failover.
And you can have only 2 hosts for Postgres - Primary and Standby. You can have 2 big Postgres machines with 64 CPUs. And 3 one-CPU hosts for Zookeper\etcd.

If you use build-in failover you have to resort to 3 big Postgres machines because you need 2/3 majority. Of course, you can install MySQL-stype arbiter - host that had no real PGDATA, only participates in voting. But this is a solution to problem induced by built-in autofailover.

Best regards, Andrey Borodin.

---

From: Kirill Reshke <[email-masked]>
Date: Wed, 16 Apr 2025 10:39:07 +0500

On Wed, 16 Apr 2025 at 10:25, Andrey Borodin <[email-masked]> wrote:
>
> I think I can provide some reasons why it cannot be neither extension, nor any part running within postmaster reign.
>
> 1. When joining cluster, there’s not PGDATA to run postmaster on top of it.

You can join the cluster on pg_basebackup of its master; So I dont get
why this is an anti-extension restriction.

> 2. After failover, old Primary node must rejoin cluster by running pg_rewind and following timeline switch.

You can run bash from extension, what's the point?

> The system in hand must be able to manipulate with PGDATA without starting Postgres.

-- 
Best regards,
Kirill Reshke

---

From: Andrey Borodin <[email-masked]>
Date: Wed, 16 Apr 2025 10:44:13 +0500

> On 16 Apr 2025, at 10:39, Kirill Reshke <[email-masked]> wrote:
> 

You cannot run bash that will stop backend running bash.

---

From: Ashutosh Bapat <[email-masked]>
Date: Wed, 16 Apr 2025 11:48:05 +0530

> Is it any better as extension than as etcd?

I feel so. An extension runs from within a postgresql process, uses
the same protocol as PostgreSQL whereas etcd is another process and
another protocol.

> I didn’t get FDW analogy. Why other distributed systems should choose Postgres extension over Zookeeper?

By other distributed systems I mean PostgreSQL distributed systems -
FDW based native sharding or native replication or a system which uses
both.

> Again, I’m lost here. Which two worlds?

Logical replication and FDW based native sharding.

> If you use build-in failover you have to resort to 3 big Postgres machines because you need 2/3 majority. Of course, you can install MySQL-stype arbiter - host that had no real PGDATA, only partici…

Users find it a waste of resources to deploy 3 big PostgreSQL
instances just for HA where 2 suffice even if they deploy 3
lightweight DCS instances. Having only some of the nodes act as DCS
and others purely PostgreSQL nodes will reduce waste of resources.

-- 
Best Wishes,
Ashutosh Bapat

---

From: Andrey Borodin <[email-masked]>
Date: Wed, 16 Apr 2025 11:27:13 +0500

> On 16 Apr 2025, at 11:18, Ashutosh Bapat <[email-masked]> wrote:
> 
> Having only some of the nodes act as DCS

But typically you need more DCS nodes than PostgreSQL nodes. Did you mean
“Having only some of nodes act as PostgreSQL and others purely DCS nodes will reduce waste of resources”?

---

From: Ashutosh Bapat <[email-masked]>
Date: Wed, 16 Apr 2025 12:15:44 +0530

> But typically you need more DCS nodes than PostgreSQL nodes. Did you mean

In a small HA setup this might be true. But not when there are many
replicas. But ...

> “Having only some nodes act as PostgreSQL and others purely DCS nodes will reduce waste of resources”?

I mean, whatever the setup may be one shouldn't require to deploy a
big PostgreSQL server just because DCS needs majority.

-- 
Best Wishes,
Ashutosh Bapat

---

From: Konstantin Osipov <[email-masked]>
Date: Wed, 16 Apr 2025 12:47:00 +0300

* Tom Lane <[email-masked]> [25/04/16 11:05]:
> Nikolay Samokhvalov <[email-masked]> writes:
> Yeah.  The two questions you'd have to get past to get this into PG
> core are:
> 
> 1. Why can't it be an extension?  (You claimed it would work more
> seamlessly in core, but I don't think you've made a proven case.)

I think this can be best addressed when the discussion moves on to
an architecture design record, where the UX and implementation
details are outlined. I'm sure there can be a lot of bike-shedding
on that part. For now I merely wanted to know if:
- maybe there is a reason this will never be accepted
- maybe someone is already working on this.

---

From: Konstantin Osipov <[email-masked]>
Date: Wed, 16 Apr 2025 12:53:09 +0300

* Ashutosh Bapat <[email-masked]> [25/04/16 11:06]:
> > My view is what Konstantin wants is automatic replication topology management. For some reason this technology is called HA, DCS, Raft, Paxos and many other scary words. But basically it manages primary_conn_info of some nodes to provide some fault-tolerance properties. I'd start to design from here, not from Raft paper.
> >

I think this experience needs to be detailed out. There are
implementations in the field that are less efficient than others.

Early etcd-raft didn't have pre-voting and had "bastardized" 
(their own definition) implementation of configuration changes
which didn't use joint consensus.

Then there is a liveness issue if leader election is implemented
in a straightforward way in large clusters. But this is addressed:
scaling up the randomized election timeout with the cluster size,
converting most of participants to non-voters in large clusters. 

Raft replication, again, if implemented in a naive way, would
require a O(outstanding transaction) * number of replicas amount of
RAM. But that doesn't have to be naive.

To sum up, I am not aware of any principal limitations in this
area.

---

From: Konstantin Osipov <[email-masked]>
Date: Wed, 16 Apr 2025 12:58:32 +0300

* Andrey Borodin <[email-masked]> [25/04/16 11:06]:
> > Andrey Borodin <[email-masked]> writes:
> >> I think it's what Konstantin is proposing. To have our own Raft implementation, without dependencies.
> > 
> > Hmm, OK.  I thought that the proposal involved relying on some existing
> > code, but re-reading the thread that was said nowhere.  Still, that
> > moves it from a large project to a really large project :-(
> > 
> > I continue to think that it'd be best to try to implement it as
> > an extension, at least up till the point of finding show-stopping
> > reasons why it cannot be that.
> 
> My question to Konstantin is Why wouldn’t you just add Raft to Patroni? Is there a reason why something like Patroni is not in core and noone rushes to get it in?
> Everyone is using it, or system like it.

Raft uses the same WAL to store configuration change records as is used
for commit records. This is at the core of the correctness of the
algorithm. This is also my biggest concern with correctness of
Patroni - but to the best of my knowledge 's 90%+ of
use cases of Patroni use a "fixed" quorum size, that's defined at
start of the deployment and never/rarely changes.
Contrast to that being able to a replica to the quorum at any
time, and all it takes is just starting this replica and pointing
it at the existing cluster. This greatly simplifies UX.

---

From: Konstantin Osipov <[email-masked]>
Date: Wed, 16 Apr 2025 13:02:00 +0300

> You cannot run bash that will stop backend running bash.

You're right there is a chicken and egg problem when you add 
Raft to an existing project, and rebootstrap 
becomes a trick, but it's a plumbing trick.

The new member needs to generate and persist a globally unique
identifier as the first step. Later it can 
reintroduce itself to the cluster given this identifier
can be preserved in the new incarnation (popen + fork).

---

From: Alastair Turner <[email-masked]>
Date: Wed, 16 Apr 2025 13:53:23 +0100

On Wed, 16 Apr 2025 at 07:18, Ashutosh Bapat <[email-masked]>
wrote:

> > If you use build-in failover you have to resort to 3 big Postgres
> machines because you need 2/3 majority. Of course, you can install
> MySQL-stype arbiter - host that had no real PGDATA, only participates in
> voting. But this is a solution to problem induced by built-in autofailover.
>
> The experience of other projects/products with automated failover based on
quorum shows that this is a critical issue for adoption. In the In-memory
Data Grid space (Coherence, Geode/GemFire) the question of how to ensure
that some nodes didn't carry any data comes up early in many architecture
discussions. When RabbitMQ shipped their Quorum Queues feature, the first
and hardest area of pushback was around all nodes hosting message content.

It's not just about the requirement for compute resources, it's also about
bandwidth and latency. Many large organisations have, for historical
reasons, pairs of data centres with very good point-to-point connectivity.
As the requirement for quorum witnesses has come up for all sorts of
things, including storage arrays, they have built arbiter/witness sites at
branches, colocation providers or even on the public cloud. More than not
holding user data or processing queries, the arbiter can't even be sent the
replication stream for the user data in the database, it just won't fit
down the pipe.

Which feels like a very difficult requirement to meet if the replication
model for all data is being changed to a quorum model.

Regards
Alastair

---

From: Konstantin Osipov <[email-masked]>
Date: Wed, 16 Apr 2025 17:07:36 +0300

> model for all data is being changed to a quorum model.

I agree master/replica deployment layouts are very popular and are
not going to directly benefit from raft. They'll still work, but no
automation will be available, just like today with Patroni.

However, if the storage cost is an argument, then the logical path is to
disaggregate storage/compute altogether, i.e. use projects like
neon.

---

From: Yura Sokolov <[email-masked]>
Date: Wed, 16 Apr 2025 17:29:09 +0300

> 2. After failover, old Primary node must rejoin cluster by running pg_rewind and following timeline switch.

It is really do-able: BiHA already does it. And BiHA runs as a child
process of postmaster, ie both postmaster and BiHA doesn't restart when
PostgreSQL needs to rewind and restart.

Yes, there are non-trivial amount of changes made into postmaster
machinery. But it is doable.

---

From: Greg Sabino Mullane <[email-masked]>
Date: Wed, 16 Apr 2025 15:29:05 -0400

On Wed, Apr 16, 2025 at 2:18 AM Ashutosh Bapat <[email-masked]>
wrote:

> Users find it a waste of resources to deploy 3 big PostgreSQL instances
> just for HA where 2 suffice even if they deploy 3 lightweight DCS
> instances. Having only some of the nodes act as DCS and others purely
> PostgreSQL nodes will reduce waste of resources.
>

A big problem is that putting your DCS into Postgres means your whole
system is now super-sensitive to IO/WAL-streaming issues, and a busy
database doing database stuff is going to start affecting the DCS stuff.
With three lightweight DCS servers, you don't really need to worry about
how stressed the database servers are. In that way, I feel etcd et al. are
adhering to the unix philosophy of "do one thing, and do it well."

Cheers,
Greg

---

From: Konstantin Osipov <[email-masked]>
Date: Wed, 16 Apr 2025 22:35:34 +0300

> database doing database stuff is going to start affecting the DCS stuff.

Affecting in what way? Do you have a scenario in mind where an
external state provider would act differently (better)?

---

From: Hannu Krosing <[email-masked]>
Date: Wed, 16 Apr 2025 23:24:09 +0200

> moves it from a large project to a really large project :-(

My understanding is that RAFT is a fancy name for what PostgreSQL is
largely already doing - "electing" a leader and then doing all the
changes through that leader (a.k.a. WAL streaming)

The thing that needs adding - and which makes it "RAFT" instead of
just a streaming replication with a failover - is what happens when
the leader goes away and one of the replicas needs to become a new
leader.

We have ways to do rollbacks and roll-forwards but the main tricky
part is "how do you know that you have not lost some changes" and here
we must have some place to store the info about at which LSN the
failover happened, so that we know to run pg_rewind if any losts hosts
come back and want to join.

And of course we need to have a way to communicate this "who is the
new leader" to clients which needs new libpgq functionality of
"failover connection pools" which hide the failovers from old clients.

The RAFT protocol could be a provider of "who is current leader info"
and optionally cache the LSN the switch happened.

> reasons why it cannot be that.

The main thing I would like to see in core is ability to do clean
*switchovers* (not failovers) by sending a magic WAL record with a
message "hey node N, please take over the write node role" over WAL so
that node N knows to self-promote and all other nodes know to start
following N starting  from the next WAL record either directly or

Why WAL - because it is already is sent to all replicas, it guarantees
continuity as it very clearly states at what LSN the write-head-switch
happens.

We also should be able to send this info to the client libraries
currently connected to the writer. so that they can choose to switch
to the new head.

The rest could be easily an extension.

Mainly we want more than one "coordinators" which can be running in
some or all of the nodes.and who agree on
- which node is current leader
- at which LSN the switch happened (so if some node coming back
discovers that it has magicall moved ahead it knows to rewind to that
LSN and then re-stream it from commonly agreed place.

 It would also be nice to have some agreed, hopefully lightweight,
notion of node identity, which we could then use for many things,
including stamping it in WAL records to guarantee / verify that all
the nodes have been on the same WAL stream all the time

But regarding weather to use RAFT I would just define a "coordinator
API" and leave it up to the specific coordinator/consensus extension
to decide how the consensus is achieved

So to summarize:

# Core should provide

- way tomove to new node,
  - for switchover a WAL-based switchover
  - for failover something similar which also writes the WAL record so
all histories are synced
- a libpq message informing clients about "new write head node"
- node IDs and more general c;luster-awareness inside the PostgreSQL
node (I had a shoutout about this in a recent pgconf.dev unconference
talk)
- a new write-node field in WAL to track write head movement
- API for a joining node to find out which cluster it joins and the
switchover history
  - in WAL it is always switchover, maybe with some info saying that
it was a forces switchover because we lost old write head
  - if some lost node comes back it may need to rewind or
re-initialize if it finds out it had been following a lost timeline
that is not fully part of

NOTE: switchovers in WAL would be very similar to timeline changes. I
am not sure how much extra info is needed there.

# Extension can provide
- agreeing on new leader node in case of failover
  - protocol can be RAFT, PAXOS or "the DBA says so" :)
- sharing fresh info about current leader and switch timelines (though
this should more likely be in core)
- anything else ???

# external apps is (likely?) needed for
- setting up cluster, provisioning machines / VMs
- setting up networking
- starting PostgreSQL servers.
- spinning up and down clients,
- communicating current leader and replica set (could be done by DNS
with agreed conventions)

---

From: Alastair Turner <[email-masked]>
Date: Wed, 16 Apr 2025 22:45:45 +0100

Hi Konstantin

On Wed, 16 Apr 2025 at 15:07, Konstantin Osipov <[email-masked]>
wrote:

> Users of Patroni and etcd setups can get automation for two-site
primary/replica pairs by putting a third etcd node on a third site. Which
only requires moving the membership/leadership data to the arbiter site,
not all database activity.

> The issue is not generally storage, but network. There may simply not be
enough bandwidth available to transmit the whole WAL to the arbiter site.

Many on-premises IT setups have this limitation in some form.

If your proposal would leave these large, traditional user organisations
(which account for thousands of Postgres HA pairs or DR pairs) doing what
they currently do with wraparound tooling like Patroni, and create a new,
in core, option for balanced 3, 5, 7... member groups, then I don't think
it's worth doing.

Regards,
Alastair

---

From: Jim Nasby <[email-masked]>
Date: Tue, 29 Apr 2025 15:16:07 -0500

I've always assumed there'd have to be at least one global stream, if for
no other purpose than to be the source of truth about transaction commit
ordering (though, I was thinking of supporting multiple streams for one
database). Presumably the same could be used for shared objects. Or perhaps
shared objects just get their own stream. Either way, having a master
commit record that points at LSNs of various other streams is what I'd been
thinking.

On Wed, Apr 23, 2025 at 12:01 PM Devrim Gündüz <[email-masked]> wrote:

> Hi,
>
> On Wed, 2025-04-23 at 11:48 -0500, Jim Nasby wrote:
> > unless we added multiple WAL streams. That would allow for splitting
> > WAL traffic across multiple devices as well as providing better
> > support for configurations that don’t replicate the entire cluster.
> > The current situation where delayed replication of a single table
> > mandates retention of all the WAL for the entire cluster is less than
> > ideal.
>
> I think the problem is handling the stream of global objects. Having
> separate stream for each database would be awesome as long as it can
> deal with the "global stream".
>
> Regards,
> --
> Devrim Gündüz
> Open Source Solution Architect, PostgreSQL Major Contributor
> BlueSky: @devrim.gunduz.org , @gunduz.org
>