Re: Parallelize WAL Sender and WAL receiver

public inbox for [email protected]  
help / color / mirror / Atom feed

From: Karthik Yellapragada <[email protected]>
To: Rui DeSousa <[email protected]>
Cc: Pgsql-admin <[email protected]>
Subject: Re: Parallelize WAL Sender and WAL receiver
Date: Wed, 4 Sep 2024 20:47:40 -0700
Message-ID: <CACXtxt-ePN=ZWPRpbYHKQXJt6H1n2X6s_K_4VcuxaPM5sWP4fA@mail.gmail.com> (raw)
In-Reply-To: <[email protected]>
References: <CACXtxt_ANB20ZWjFQt=BOtCN50g6MvNvQPokv5TwPSwwXchdkw@mail.gmail.com>
	<[email protected]>

Appreciate the responses, the network is good.. around 350 MB/s between
prinary and secondary, both primary and secondary in the same DC.  And it’s
ASYC replication..

I see the issue when the writes are super heavy ,
I am only thinking if we can add more processes that sends / receives the
WALs , we can speed up the transfer rate of WALs to the secondary..

I understand the Apply can not be parallelized..

On Wed, Sep 4, 2024 at 6:52 PM Rui DeSousa <[email protected]> wrote:

>
>
> > On Sep 4, 2024, at 9:03 PM, Karthik Yellapragada <
> [email protected]> wrote:
> >
> > Hello all,
> >
> > I frequently face the problem of wals generated faster than the wals
> transferred and applied.
> >
> > Is there a way to speed up the process? So I don’t accumulate a lot of
> WALs at the primary?
> >
> > Regards,
> > Karthik.
>
> What do you mean by parallelized WAL sender? Do you have many replicas
> connecting to the primary? Have you tried cascading replication?
>
> What’s the network between the systems? Is a WAN with high latency? Can
> the replica pull from the WAL file from an archive?  For a high latency
> networks like New York to London; I would terminate the WAL receiver via a
> script if replication exceeded an acceptable delay to ensure SLA  where
> met.  It is faster to pull the WAL files from the local archive which where
> already replicated to London than streaming replication over a high latency
> WAN.  After It catches up, it will reconnect to streaming replication.
>
> --
Thanks & Regards
Karthik Yellapragada
+1 860 830 5235

view thread (10+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected]
  Subject: Re: Parallelize WAL Sender and WAL receiver
  In-Reply-To: <CACXtxt-ePN=ZWPRpbYHKQXJt6H1n2X6s_K_4VcuxaPM5sWP4fA@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox