pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Jim Nasby <jim.nasby@gmail.com>
To: Antonin Bas <antonin.bas@gmail.com>
To: pgsql-general@lists.postgresql.org
Subject: Re: What should I expect when creating many logical replication slots?
Date: Tue, 16 Jan 2024 12:51:44 -0600
Message-ID: <7e8f8a71-c7dc-4625-9d96-5e4f81540e01@gmail.com> (raw)
In-Reply-To: <CAAkB0aDof-atNom4qO_RGefgPDib3ukEzX1B9Tva11nusWMriA@mail.gmail.com>
References: <CAAkB0aDof-atNom4qO_RGefgPDib3ukEzX1B9Tva11nusWMriA@mail.gmail.com>

On 1/11/24 6:17 PM, Antonin Bas wrote:
> Hi all,
> 
> I have a use case for which I am considering using Postgres Logical 
> Replication, but I would like to scale up to 100 or even 200 
> replication slots.
> 
> I have increased max_wal_senders and max_replication_slots to 100 (also 
> making sure that max_connections is large enough). Things seem to be 
> working pretty well so far based on some PoC code I have written. 
> Postgres is creating a walsender process for each replication slot, as 
> expected, and the memory footprint of each one is around 4MB.
> 
> So I am quite happy with the way things are working, but I am a bit 
> uneasy about increasing these configuration values by 10-20x compared to 
> their defaults (both max_wal_senders and max_replication_slots default 
> to 10).
> 
> Is there anything I should be looking out for specifically? Is it 
> considered an anti-pattern to use that many replication slots and 
> walsender processes? And, when my database comes under heavy write load, 
> will walsender processes start consuming a large amount of CPU / memory 
> (I recognize that this is a vague question, I am still working on some 
> empirical testing).

The biggest issue with logical decoding (what drives logical 
replication) is that every subscriber has to completely decode 
everything for it's publication, which can be extremely memory intensive 
under certain circumstances (long running transacitons being one 
potential trigger). Decoders also have to read through all WAL traffic, 
regardless of what their publication is set to - everything runs of the 
single WAL stream.

Note that this only applies to actually decoding - simply having a large 
number of slots isn't much of an issue. Even having a large number of 
subscribers that aren't consuming isn't a resource issue (though it IS 
an issue for MVCC / vacuuming!) - to test you need to have all the 
decoders that you expect to support.

Ultimately, I'd be concerned with trying to support 100+ slots unless 
you know that your change rate isn't super high and that you don't have 
long-running transactions.
-- 
Jim Nasby, Data Architect, Austin TX






view thread (4+ messages)  latest in thread

Message-ID: <7e8f8a71-c7dc-4625-9d96-5e4f81540e01@gmail.com>
Permalink:  ../7e8f8a71-c7dc-4625-9d96-5e4f81540e01@gmail.com/
Also on:    postgresql.org/message-id/7e8f8a71-c7dc-4625-9d96-5e4f81540e01@gmail.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-general@postgresql.org
  Cc: jim.nasby@gmail.com, antonin.bas@gmail.com
  Subject: Re: What should I expect when creating many logical replication slots?
  In-Reply-To: <7e8f8a71-c7dc-4625-9d96-5e4f81540e01@gmail.com>
  References: <CAAkB0aDof-atNom4qO_RGefgPDib3ukEzX1B9Tva11nusWMriA@mail.gmail.com> <7e8f8a71-c7dc-4625-9d96-5e4f81540e01@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox