Subject: Parallelize WAL Sender and WAL receiver

From: Karthik Yellapragada <[email-masked]>
Date: Wed, 4 Sep 2024 18:03:57 -0700

Hello all,

I frequently face the problem of wals generated faster than the wals
transferred and applied.

Is there a way to speed up the process? So I don’t accumulate a lot of WALs
at the primary?

Regards,
Karthik.
-- 
Thanks & Regards
Karthik Yellapragada
+1 860 830 5235

---

From: Ron Johnson <[email-masked]>
Date: Wed, 4 Sep 2024 21:34:03 -0400

On Wed, Sep 4, 2024 at 9:04 PM Karthik Yellapragada <
[email-masked]> wrote:

> Is there a way to speed up the process? So I don’t accumulate a lot of
> WALs at the primary?
>

Accumulation on the *primary* sounds like slow (or not-fast-enough)
networking.
Or using synchronous replication instead of async replication.
Or a combination of both.

-- 
Death to America, and butter sauce.
Iraq lobster!

---

From: Keith <[email-masked]>
Date: Wed, 4 Sep 2024 21:41:37 -0400

> Is there a way to speed up the process? So I don’t accumulate a lot of

Take a look at pgBackRest. It provides an option for asynchronous WAL
archiving and retrieval. The actual replay of the WAL is always single
threaded.

https://pgbackrest.org/

Keith

---

From: Rui DeSousa <[email-masked]>
Date: Wed, 4 Sep 2024 21:52:01 -0400

> On Sep 4, 2024, at 9:03 PM, Karthik Yellapragada <[email-masked]> wrote:
> 
> Hello all,
> 
> I frequently face the problem of wals generated faster than the wals transferred and applied.
> 
> Is there a way to speed up the process? So I don’t accumulate a lot of WALs at the primary?
> 
> Regards,
> Karthik.

What do you mean by parallelized WAL sender? Do you have many replicas connecting to the primary? Have you tried cascading replication?

What’s the network between the systems? Is a WAN with high latency? Can the replica pull from the WAL file from an archive?  For a high latency networks like New York to London; I would terminate the WAL receiver via a script if replication exceeded an acceptable delay to ensure SLA  where met.  It is faster to pull the WAL files from the local archive which where already replicated to London than streaming replication over a high latency WAN.  After It catches up, it will reconnect to streaming replication.

---

From: Karthik Yellapragada <[email-masked]>
Date: Wed, 4 Sep 2024 20:47:40 -0700

Appreciate the responses, the network is good.. around 350 MB/s between
prinary and secondary, both primary and secondary in the same DC.  And it’s
ASYC replication..

I see the issue when the writes are super heavy ,
I am only thinking if we can add more processes that sends / receives the
WALs , we can speed up the transfer rate of WALs to the secondary..

I understand the Apply can not be parallelized..

On Wed, Sep 4, 2024 at 6:52 PM Rui DeSousa <[email-masked]> wrote:

> What do you mean by parallelized WAL sender? Do you have many replicas
> connecting to the primary? Have you tried cascading replication?
>
> What’s the network between the systems? Is a WAN with high latency? Can
> the replica pull from the WAL file from an archive?  For a high latency
> networks like New York to London; I would terminate the WAL receiver via a
> script if replication exceeded an acceptable delay to ensure SLA  where
> met.  It is faster to pull the WAL files from the local archive which where
> already replicated to London than streaming replication over a high latency
> WAN.  After It catches up, it will reconnect to streaming replication.
>
> --
Thanks & Regards
Karthik Yellapragada
+1 860 830 5235

---

From: Rui DeSousa <[email-masked]>
Date: Thu, 5 Sep 2024 00:33:10 -0400

> On Sep 4, 2024, at 11:47 PM, Karthik Yellapragada <[email-masked]> wrote:
> 
> Appreciate the responses, the network is good.. around 350 MB/s between prinary and secondary, both primary and secondary in the same DC.  And it’s ASYC replication.. 
> 
> I am only thinking if we can add more processes that sends / receives the WALs , we can speed up the transfer rate of WALs to the secondary..
> 

Sounds like you have 10GbE between the nodes? Are you sure it’s send rate? If so, I would look at the disk subsystem.

---

From: ajit wangkhem <[email-masked]>
Date: Thu, 5 Sep 2024 13:24:48 +0530

If speed create problem than make 3 easy script. 1. compress wal_archive.
2. ship wal archive over network and than 3. in the secondary server just
decompress it.

---

From: Scott Ribe <[email-masked]>
Date: Thu, 5 Sep 2024 06:08:36 -0600

> On Sep 5, 2024, at 1:54 AM, ajit wangkhem <[email-masked]> wrote:
> 
> If speed create problem than make 3 easy script. 1. compress wal_archive. 2. ship wal archive over network and than 3. in the secondary server just decompress it. 

I'm guessing you've never benchmarked your compression? With 10GbE, the CPU overhead of compression will just slow it down.

---

From: Muhammad Usman Khan <[email-masked]>
Date: Fri, 6 Sep 2024 08:33:23 +0500

Hi,
You can try to set the following parameters in the conf file which may help
to overcome this problem.

wal_sender_timeout = 60s
wal_receiver_status_interval = 1s
Ensure you have sufficient WAL senders  like max_wal_senders = 10
max_wal_size = 2GB
min_wal_size = 80MB
wal_buffers = 16MB
wal_level = replica

---

From: Laurenz Albe <[email-masked]>
Date: Fri, 06 Sep 2024 07:02:40 +0200

On Fri, 2024-09-06 at 08:33 +0500, Muhammad Usman Khan wrote:
> On Thu, 5 Sept 2024 at 06:04, Karthik Yellapragada <[email-masked]> wrote:
> You can try to set the following parameters in the conf file which may help to overcome this problem.
>
> wal_sender_timeout = 60s

These settings won't do anything to mitigate the problem.

All you can do is get more network bandwidth.

Yours,
Laurenz Albe