Re: Parallelize WAL Sender and WAL receiver

public inbox for [email protected]  
help / color / mirror / Atom feed

From: ajit wangkhem <[email protected]>
To: Karthik Yellapragada <[email protected]>
Cc: Rui DeSousa <[email protected]>
Cc: Pgsql-admin <[email protected]>
Subject: Re: Parallelize WAL Sender and WAL receiver
Date: Thu, 5 Sep 2024 13:24:48 +0530
Message-ID: <CAMurXp2np_xKSFcsuqDmpw+O-cwN3KMBOZdN4Ksfp58=AbtWwA@mail.gmail.com> (raw)
In-Reply-To: <CACXtxt-ePN=ZWPRpbYHKQXJt6H1n2X6s_K_4VcuxaPM5sWP4fA@mail.gmail.com>
References: <CACXtxt_ANB20ZWjFQt=BOtCN50g6MvNvQPokv5TwPSwwXchdkw@mail.gmail.com>
	<[email protected]>
	<CACXtxt-ePN=ZWPRpbYHKQXJt6H1n2X6s_K_4VcuxaPM5sWP4fA@mail.gmail.com>

If speed create problem than make 3 easy script. 1. compress wal_archive.
2. ship wal archive over network and than 3. in the secondary server just
decompress it.

On Thu, Sep 5, 2024 at 9:18 AM Karthik Yellapragada <
[email protected]> wrote:

> Appreciate the responses, the network is good.. around 350 MB/s between
> prinary and secondary, both primary and secondary in the same DC.  And it’s
> ASYC replication..
>
> I see the issue when the writes are super heavy ,
> I am only thinking if we can add more processes that sends / receives the
> WALs , we can speed up the transfer rate of WALs to the secondary..
>
> I understand the Apply can not be parallelized..
>
> On Wed, Sep 4, 2024 at 6:52 PM Rui DeSousa <[email protected]> wrote:
>
>>
>>
>> > On Sep 4, 2024, at 9:03 PM, Karthik Yellapragada <
>> [email protected]> wrote:
>> >
>> > Hello all,
>> >
>> > I frequently face the problem of wals generated faster than the wals
>> transferred and applied.
>> >
>> > Is there a way to speed up the process? So I don’t accumulate a lot of
>> WALs at the primary?
>> >
>> > Regards,
>> > Karthik.
>>
>> What do you mean by parallelized WAL sender? Do you have many replicas
>> connecting to the primary? Have you tried cascading replication?
>>
>> What’s the network between the systems? Is a WAN with high latency? Can
>> the replica pull from the WAL file from an archive?  For a high latency
>> networks like New York to London; I would terminate the WAL receiver via a
>> script if replication exceeded an acceptable delay to ensure SLA  where
>> met.  It is faster to pull the WAL files from the local archive which where
>> already replicated to London than streaming replication over a high latency
>> WAN.  After It catches up, it will reconnect to streaming replication.
>>
>> --
> Thanks & Regards
> Karthik Yellapragada
> +1 860 830 5235
>

view thread (10+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Parallelize WAL Sender and WAL receiver
  In-Reply-To: <CAMurXp2np_xKSFcsuqDmpw+O-cwN3KMBOZdN4Ksfp58=AbtWwA@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox