From: Klaus Darilion <klaus.mailinglists@pernau.at>
To: Jim Nasby <jim.nasby@gmail.com>
Cc: Antonin Bas <antonin.bas@gmail.com>
Cc: pgsql-general@lists.postgresql.org
Subject: Re: What should I expect when creating many logical replication slots?
Date: Tue, 30 Jan 2024 00:01:26 +0100
Message-ID: <8d3be16df9f21a71c94c0ce0d7ebe553@pernau.at> (raw)
In-Reply-To: <7e8f8a71-c7dc-4625-9d96-5e4f81540e01@gmail.com>
References: <CAAkB0aDof-atNom4qO_RGefgPDib3ukEzX1B9Tva11nusWMriA@mail.gmail.com>
<7e8f8a71-c7dc-4625-9d96-5e4f81540e01@gmail.com>
Am 2024-01-16 19:51, schrieb Jim Nasby:
> On 1/11/24 6:17 PM, Antonin Bas wrote:
>> Hi all,
>>
>> I have a use case for which I am considering using Postgres Logical
>> Replication, but I would like to scale up to 100 or even 200
>> replication slots.
>>
>> I have increased max_wal_senders and max_replication_slots to 100
>> (also making sure that max_connections is large enough). Things seem
>> to be working pretty well so far based on some PoC code I have
>> written. Postgres is creating a walsender process for each replication
>> slot, as expected, and the memory footprint of each one is around 4MB.
>>
>> So I am quite happy with the way things are working, but I am a bit
>> uneasy about increasing these configuration values by 10-20x compared
>> to their defaults (both max_wal_senders and max_replication_slots
>> default to 10).
>>
>> Is there anything I should be looking out for specifically? Is it
>> considered an anti-pattern to use that many replication slots and
>> walsender processes? And, when my database comes under heavy write
>> load, will walsender processes start consuming a large amount of CPU /
>> memory (I recognize that this is a vague question, I am still working
>> on some empirical testing).
>
> The biggest issue with logical decoding (what drives logical
> replication) is that every subscriber has to completely decode
> everything for it's publication, which can be extremely memory
> intensive under certain circumstances (long running transacitons being
> one potential trigger). Decoders also have to read through all WAL
> traffic, regardless of what their publication is set to - everything
> runs of the single WAL stream.
Indeed, big/long-running transaction can be an issue. Today I added a
column with SERIAL to a table with 55 mio rows. Although that table is
not in the publication, the hugh transaction got written to the WAL and
spilled to disk by the WAL senders. Having 50 WAL senders, we basically
created a DoS-amplification-attack against our disk. Luckily we could
increase the disk size.*
Besides that, our DB has (in my point of view) plenty of UPDATE/INSERTs
and did not have replication performance problems with currently 50
replication slots. But if that would become a bottleneck, we would use
cascading replication, ie the master uses logical replication to 2
"distribution" servers, which further replicates to the 50+ subscribers.
That way, the inbetween "distribution" server would also filter out WAL
decoding for changes that are not part of the publication.
regards
Klaus
* Does somebody know why the WAL sender has to track transactions that
make changes to tables which are not the publication?
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: klaus.mailinglists@pernau.at, jim.nasby@gmail.com, antonin.bas@gmail.com
Subject: Re: What should I expect when creating many logical replication slots?
In-Reply-To: <8d3be16df9f21a71c94c0ce0d7ebe553@pernau.at>
References: <CAAkB0aDof-atNom4qO_RGefgPDib3ukEzX1B9Tva11nusWMriA@mail.gmail.com> <7e8f8a71-c7dc-4625-9d96-5e4f81540e01@gmail.com> <8d3be16df9f21a71c94c0ce0d7ebe553@pernau.at>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox