Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1u5Cre-00Du8s-8f for pgsql-hackers@arkaria.postgresql.org; Thu, 17 Apr 2025 00:12:23 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1u5Crc-0099LE-Iv for pgsql-hackers@arkaria.postgresql.org; Thu, 17 Apr 2025 00:12:21 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1u5Crc-0099L3-4c for pgsql-hackers@lists.postgresql.org; Thu, 17 Apr 2025 00:12:21 +0000 Received: from mail-ej1-x62e.google.com ([2a00:1450:4864:20::62e]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1u5CrY-000Tj0-1f for pgsql-hackers@lists.postgresql.org; Thu, 17 Apr 2025 00:12:20 +0000 Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-ac3b12e8518so38967866b.0 for ; Wed, 16 Apr 2025 17:12:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cjt-london.20230601.gappssmtp.com; s=20230601; t=1744848736; x=1745453536; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=aqfNEmrYVWKoAFJRXpHmY9HbzMeTX4UxVZPfw+fw1Pw=; b=lbZHY1SpRTp6TvqyDnCnh9YQ/jfP4GyGuPZA/liXDdqvMIDosUK+mhAXF1gx9RlO6m 7BbL/kpTEH1A3Us7TXH7B2YrD/rx3VFljWmvb8vvsUd1RDk9yD38rDS5+l1Eg9tD3Ep8 cCBpZmq+TVWOed9Eh5cIADNIRjJeyGIKAAYrrhQsyHJWiyXZGGxbdzInP9PNlhO5KMBJ TXsKe8kMODjq++zKkIxD5tYg/q3HcLsA4ftveIIoQD9OLN4Mt27r1slfPNUsjj8fZNi1 OWVrwPdYjDopGzqg6Gar9Y4ct7SkfBRL8tA/D/ejFwvsP1Yb4OJNydmETzV2jFj8vahY QABw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744848736; x=1745453536; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=aqfNEmrYVWKoAFJRXpHmY9HbzMeTX4UxVZPfw+fw1Pw=; b=UHbOsuNxAFq5DVHAsONW9L1cAMLivgUSE/xAGRmVhHYLvG4HaM5uekd/M+BpX4U/89 3+QbtA3M2H/w/bkim0NMLm5GmK9nEY+QkCCkXKeWlXJhSzDCzusiUeBR7fPXeTySA3j4 ZQU7FbJ+dKiaLAi+KmUtxvpbgWT8EuFGCbI1yRgJzn6kB43Lr5ie77D5mlsOV0OSYQM2 utusYh3C9OxHc7JRw1lnGcX+Mram5EpNSvCjI8sBsZMprzYUcTSVxxFrjWBDXTjTBSoo RaQRX+LAN3Ov0Jnonz/KLhGq/eWNWDa35FJ1/GjRLLfrbxn9ZAjKtam1+dXdTkUtDh8n J4DA== X-Forwarded-Encrypted: i=1; AJvYcCVvlq47vjPcikBA8R+W0yYaVu8OyNSgpgvLMYlZQzMSC8sBS6ZokassGIv8PuGl6QAnbfBjonHNsf2ejJzE@lists.postgresql.org X-Gm-Message-State: AOJu0YwEkhZ5pRPbB1LetgwAyXNE0hsbMIdrjNbnzg1UpVkCdWqKBggY kly217zxHN1bOSqvL82r5Xo9fJbcxsyVrKeL0zh2tjg1vqd4e7HAikQxJH3GMqbOs1fP5yKsuxX sOtd3jdosO8hW0RD2QQvzTi7eePWdtUqllWmpoYDsEh5iHZXwVwag4A== X-Gm-Gg: ASbGnctPkLlsCrwZzDh9y0Ev+kNHUbBdTZjCk0FQ9p0lgzAmcTXNPC8I++eCZpUysoy 0F48ka0XTibUVfkTyQ/Q4l94o35JHoZEzYDPB4Xc943tKpdIpZffffdyO/iUliyGoFE0R/RdHD6 1YT2qz75l1po2KlUNeeOnl X-Google-Smtp-Source: AGHT+IGCE1vnRKPbpOdaOnR5tT5AJYt+dqGvZIjMHBWHVBIdRQX9TGx9NgrbauAo2oNE3gwWzWZn04NpnFo6Trd0XCQ= X-Received: by 2002:a17:907:930c:b0:ac3:8895:2776 with SMTP id a640c23a62f3a-acb4287676cmr354564966b.5.1744839955974; Wed, 16 Apr 2025 14:45:55 -0700 (PDT) MIME-Version: 1.0 References: <1798838.1744759182@sss.pgh.pa.us> <20FB597F-641F-48F8-8428-D8DDBA802D58@yandex-team.ru> <212D5973-FDD0-4CF5-BCD0-2760EC319DF3@yandex-team.ru> In-Reply-To: From: Alastair Turner Date: Wed, 16 Apr 2025 22:45:45 +0100 X-Gm-Features: ATxdqUH6THjbYU6gUhz3RDUShrcm3zIP6Di3Nk_1xskslH3E52aJ-v_MSS7Em8c Message-ID: Subject: Re: Built-in Raft replication To: Konstantin Osipov Cc: Ashutosh Bapat , Andrey Borodin , Tom Lane , Greg Sabino Mullane , Nikolay Samokhvalov , PostgreSQL Hackers Content-Type: multipart/alternative; boundary="000000000000c427ae0632ec359c" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000c427ae0632ec359c Content-Type: text/plain; charset="UTF-8" Hi Konstantin On Wed, 16 Apr 2025 at 15:07, Konstantin Osipov wrote: > * Alastair Turner [25/04/16 15:58]: > > > > > If you use build-in failover you have to resort to 3 big Postgres > > > machines because you need 2/3 majority. Of course, you can install > > > MySQL-stype arbiter - host that had no real PGDATA, only participates > in > > > voting. But this is a solution to problem induced by built-in > autofailover. > > > > > > Users find it a waste of resources to deploy 3 big PostgreSQL > > > instances just for HA where 2 suffice even if they deploy 3 > > > lightweight DCS instances. Having only some of the nodes act as DCS > > > and others purely PostgreSQL nodes will reduce waste of resources. > > > > > > The experience of other projects/products with automated failover > based on > > quorum shows that this is a critical issue for adoption. In the In-memory > > Data Grid space (Coherence, Geode/GemFire) the question of how to ensure > > that some nodes didn't carry any data comes up early in many architecture > > discussions. When RabbitMQ shipped their Quorum Queues feature, the first > > and hardest area of pushback was around all nodes hosting message > content. > > > > It's not just about the requirement for compute resources, it's also > about > > bandwidth and latency. Many large organisations have, for historical > > reasons, pairs of data centres with very good point-to-point > connectivity. > > As the requirement for quorum witnesses has come up for all sorts of > > things, including storage arrays, they have built arbiter/witness sites > at > > branches, colocation providers or even on the public cloud. More than not > > holding user data or processing queries, the arbiter can't even be sent > the > > replication stream for the user data in the database, it just won't fit > > down the pipe. > > > > Which feels like a very difficult requirement to meet if the replication > > model for all data is being changed to a quorum model. > > I agree master/replica deployment layouts are very popular and are > not going to directly benefit from raft. They'll still work, but no > automation will be available, just like today with Patroni. > > Users of Patroni and etcd setups can get automation for two-site primary/replica pairs by putting a third etcd node on a third site. Which only requires moving the membership/leadership data to the arbiter site, not all database activity. > However, if the storage cost is an argument, then the logical path is to > disaggregate storage/compute altogether, i.e. use projects like > neon. > > The issue is not generally storage, but network. There may simply not be enough bandwidth available to transmit the whole WAL to the arbiter site. Many on-premises IT setups have this limitation in some form. If your proposal would leave these large, traditional user organisations (which account for thousands of Postgres HA pairs or DR pairs) doing what they currently do with wraparound tooling like Patroni, and create a new, in core, option for balanced 3, 5, 7... member groups, then I don't think it's worth doing. Regards, Alastair --000000000000c427ae0632ec359c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Konstantin
=
= On Wed, 16 Apr 2025 at 15:07, Konstantin Osipov <kostja.osipov@gmail.com> wrote= :
* Alastair Tur= ner <minion@dec= odable.me> [25/04/16 15:58]:

> > > If you use build-in failover you have to resort to 3 big Pos= tgres
> > machines because you need 2/3 majority. Of course, you can instal= l
> > MySQL-stype arbiter - host that had no real PGDATA, only particip= ates in
> > voting. But this is a solution to problem induced by built-in aut= ofailover.
> >
> > Users find it a waste of resources to deploy 3 big PostgreSQL
> > instances just for HA where 2 suffice even if they deploy 3
> > lightweight DCS instances. Having only some of the nodes act as D= CS
> > and others purely PostgreSQL nodes will reduce waste of resources= .
> >
> > The experience of other projects/products with automated failover= based on
> quorum shows that this is a critical issue for adoption. In the In-mem= ory
> Data Grid space (Coherence, Geode/GemFire) the question of how to ensu= re
> that some nodes didn't carry any data comes up early in many archi= tecture
> discussions. When RabbitMQ shipped their Quorum Queues feature, the fi= rst
> and hardest area of pushback was around all nodes hosting message cont= ent.
>
> It's not just about the requirement for compute resources, it'= s also about
> bandwidth and latency. Many large organisations have, for historical > reasons, pairs of data centres with very good point-to-point connectiv= ity.
> As the requirement for quorum witnesses has come up for all sorts of > things, including storage arrays, they have built arbiter/witness site= s at
> branches, colocation providers or even on the public cloud. More than = not
> holding user data or processing queries, the arbiter can't even be= sent the
> replication stream for the user data in the database, it just won'= t fit
> down the pipe.
>
> Which feels like a very difficult requirement to meet if the replicati= on
> model for all data is being changed to a quorum model.

I agree master/replica deployment layouts are very popular and are
not going to directly benefit from raft. They'll still work, but no
automation will be available, just like today with Patroni.

Users of Patroni and etcd setups can get automation f= or two-site primary/replica pairs by putting a third etcd node on a third s= ite. Which only requires moving the membership/leadership data to the arbit= er site, not all database activity.
=C2=A0
However, if the storage cost is an argument, then the logical path is to disaggregate storage/compute altogether, i.e. use projects like
neon.

The issue is not generally storage, but network. Ther= e may simply not be enough bandwidth available to transmit the whole WAL to= the arbiter site.

Many on-premises IT setups have= this limitation in some form.

If your proposal wo= uld leave these large, traditional user organisations (which account for th= ousands of Postgres HA pairs or DR pairs) doing what they currently do with= wraparound tooling like Patroni, and create a new, in core, option for bal= anced 3, 5, 7... member groups, then I don't think it's worth doing= .

Regards,
Alastair
--000000000000c427ae0632ec359c--