public inbox for [email protected]
help / color / mirror / Atom feedParallelize WAL Sender and WAL receiver
10+ messages / 8 participants
[nested] [flat]
* Parallelize WAL Sender and WAL receiver
@ 2024-09-05 01:03 Karthik Yellapragada <[email protected]>
2024-09-05 01:34 ` Re: Parallelize WAL Sender and WAL receiver Ron Johnson <[email protected]>
2024-09-05 01:41 ` Re: Parallelize WAL Sender and WAL receiver Keith <[email protected]>
2024-09-05 01:52 ` Re: Parallelize WAL Sender and WAL receiver Rui DeSousa <[email protected]>
2024-09-06 03:33 ` Re: Parallelize WAL Sender and WAL receiver Muhammad Usman Khan <[email protected]>
0 siblings, 4 replies; 10+ messages in thread
From: Karthik Yellapragada @ 2024-09-05 01:03 UTC (permalink / raw)
To: Pgsql-admin <[email protected]>
Hello all,
I frequently face the problem of wals generated faster than the wals
transferred and applied.
Is there a way to speed up the process? So I don’t accumulate a lot of WALs
at the primary?
Regards,
Karthik.
--
Thanks & Regards
Karthik Yellapragada
+1 860 830 5235
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Parallelize WAL Sender and WAL receiver
2024-09-05 01:03 Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
@ 2024-09-05 01:34 ` Ron Johnson <[email protected]>
3 siblings, 0 replies; 10+ messages in thread
From: Ron Johnson @ 2024-09-05 01:34 UTC (permalink / raw)
To: Pgsql-admin <[email protected]>
On Wed, Sep 4, 2024 at 9:04 PM Karthik Yellapragada <
[email protected]> wrote:
> Hello all,
>
> I frequently face the problem of wals generated faster than the wals
> transferred and applied.
>
> Is there a way to speed up the process? So I don’t accumulate a lot of
> WALs at the primary?
>
Accumulation on the *primary* sounds like slow (or not-fast-enough)
networking.
Or using synchronous replication instead of async replication.
Or a combination of both.
--
Death to America, and butter sauce.
Iraq lobster!
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Parallelize WAL Sender and WAL receiver
2024-09-05 01:03 Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
@ 2024-09-05 01:41 ` Keith <[email protected]>
3 siblings, 0 replies; 10+ messages in thread
From: Keith @ 2024-09-05 01:41 UTC (permalink / raw)
To: Karthik Yellapragada <[email protected]>; +Cc: Pgsql-admin <[email protected]>
On Wed, Sep 4, 2024 at 9:04 PM Karthik Yellapragada <
[email protected]> wrote:
> Hello all,
>
> I frequently face the problem of wals generated faster than the wals
> transferred and applied.
>
> Is there a way to speed up the process? So I don’t accumulate a lot of
> WALs at the primary?
>
> Regards,
> Karthik.
> --
> Thanks & Regards
> Karthik Yellapragada
> +1 860 830 5235
>
Take a look at pgBackRest. It provides an option for asynchronous WAL
archiving and retrieval. The actual replay of the WAL is always single
threaded.
https://pgbackrest.org/
Keith
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Parallelize WAL Sender and WAL receiver
2024-09-05 01:03 Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
@ 2024-09-05 01:52 ` Rui DeSousa <[email protected]>
2024-09-05 03:47 ` Re: Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
3 siblings, 1 reply; 10+ messages in thread
From: Rui DeSousa @ 2024-09-05 01:52 UTC (permalink / raw)
To: Karthik Yellapragada <[email protected]>; +Cc: Pgsql-admin <[email protected]>
> On Sep 4, 2024, at 9:03 PM, Karthik Yellapragada <[email protected]> wrote:
>
> Hello all,
>
> I frequently face the problem of wals generated faster than the wals transferred and applied.
>
> Is there a way to speed up the process? So I don’t accumulate a lot of WALs at the primary?
>
> Regards,
> Karthik.
What do you mean by parallelized WAL sender? Do you have many replicas connecting to the primary? Have you tried cascading replication?
What’s the network between the systems? Is a WAN with high latency? Can the replica pull from the WAL file from an archive? For a high latency networks like New York to London; I would terminate the WAL receiver via a script if replication exceeded an acceptable delay to ensure SLA where met. It is faster to pull the WAL files from the local archive which where already replicated to London than streaming replication over a high latency WAN. After It catches up, it will reconnect to streaming replication.
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Parallelize WAL Sender and WAL receiver
2024-09-05 01:03 Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
2024-09-05 01:52 ` Re: Parallelize WAL Sender and WAL receiver Rui DeSousa <[email protected]>
@ 2024-09-05 03:47 ` Karthik Yellapragada <[email protected]>
2024-09-05 04:33 ` Re: Parallelize WAL Sender and WAL receiver Rui DeSousa <[email protected]>
2024-09-05 07:54 ` Re: Parallelize WAL Sender and WAL receiver ajit wangkhem <[email protected]>
0 siblings, 2 replies; 10+ messages in thread
From: Karthik Yellapragada @ 2024-09-05 03:47 UTC (permalink / raw)
To: Rui DeSousa <[email protected]>; +Cc: Pgsql-admin <[email protected]>
Appreciate the responses, the network is good.. around 350 MB/s between
prinary and secondary, both primary and secondary in the same DC. And it’s
ASYC replication..
I see the issue when the writes are super heavy ,
I am only thinking if we can add more processes that sends / receives the
WALs , we can speed up the transfer rate of WALs to the secondary..
I understand the Apply can not be parallelized..
On Wed, Sep 4, 2024 at 6:52 PM Rui DeSousa <[email protected]> wrote:
>
>
> > On Sep 4, 2024, at 9:03 PM, Karthik Yellapragada <
> [email protected]> wrote:
> >
> > Hello all,
> >
> > I frequently face the problem of wals generated faster than the wals
> transferred and applied.
> >
> > Is there a way to speed up the process? So I don’t accumulate a lot of
> WALs at the primary?
> >
> > Regards,
> > Karthik.
>
> What do you mean by parallelized WAL sender? Do you have many replicas
> connecting to the primary? Have you tried cascading replication?
>
> What’s the network between the systems? Is a WAN with high latency? Can
> the replica pull from the WAL file from an archive? For a high latency
> networks like New York to London; I would terminate the WAL receiver via a
> script if replication exceeded an acceptable delay to ensure SLA where
> met. It is faster to pull the WAL files from the local archive which where
> already replicated to London than streaming replication over a high latency
> WAN. After It catches up, it will reconnect to streaming replication.
>
> --
Thanks & Regards
Karthik Yellapragada
+1 860 830 5235
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Parallelize WAL Sender and WAL receiver
2024-09-05 01:03 Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
2024-09-05 01:52 ` Re: Parallelize WAL Sender and WAL receiver Rui DeSousa <[email protected]>
2024-09-05 03:47 ` Re: Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
@ 2024-09-05 04:33 ` Rui DeSousa <[email protected]>
1 sibling, 0 replies; 10+ messages in thread
From: Rui DeSousa @ 2024-09-05 04:33 UTC (permalink / raw)
To: Karthik Yellapragada <[email protected]>; +Cc: Pgsql-admin <[email protected]>
> On Sep 4, 2024, at 11:47 PM, Karthik Yellapragada <[email protected]> wrote:
>
> Appreciate the responses, the network is good.. around 350 MB/s between prinary and secondary, both primary and secondary in the same DC. And it’s ASYC replication..
>
> I see the issue when the writes are super heavy ,
> I am only thinking if we can add more processes that sends / receives the WALs , we can speed up the transfer rate of WALs to the secondary..
>
> I understand the Apply can not be parallelized..
Sounds like you have 10GbE between the nodes? Are you sure it’s send rate? If so, I would look at the disk subsystem.
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Parallelize WAL Sender and WAL receiver
2024-09-05 01:03 Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
2024-09-05 01:52 ` Re: Parallelize WAL Sender and WAL receiver Rui DeSousa <[email protected]>
2024-09-05 03:47 ` Re: Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
@ 2024-09-05 07:54 ` ajit wangkhem <[email protected]>
2024-09-05 12:08 ` Re: Parallelize WAL Sender and WAL receiver Scott Ribe <[email protected]>
1 sibling, 1 reply; 10+ messages in thread
From: ajit wangkhem @ 2024-09-05 07:54 UTC (permalink / raw)
To: Karthik Yellapragada <[email protected]>; +Cc: Rui DeSousa <[email protected]>; Pgsql-admin <[email protected]>
If speed create problem than make 3 easy script. 1. compress wal_archive.
2. ship wal archive over network and than 3. in the secondary server just
decompress it.
On Thu, Sep 5, 2024 at 9:18 AM Karthik Yellapragada <
[email protected]> wrote:
> Appreciate the responses, the network is good.. around 350 MB/s between
> prinary and secondary, both primary and secondary in the same DC. And it’s
> ASYC replication..
>
> I see the issue when the writes are super heavy ,
> I am only thinking if we can add more processes that sends / receives the
> WALs , we can speed up the transfer rate of WALs to the secondary..
>
> I understand the Apply can not be parallelized..
>
> On Wed, Sep 4, 2024 at 6:52 PM Rui DeSousa <[email protected]> wrote:
>
>>
>>
>> > On Sep 4, 2024, at 9:03 PM, Karthik Yellapragada <
>> [email protected]> wrote:
>> >
>> > Hello all,
>> >
>> > I frequently face the problem of wals generated faster than the wals
>> transferred and applied.
>> >
>> > Is there a way to speed up the process? So I don’t accumulate a lot of
>> WALs at the primary?
>> >
>> > Regards,
>> > Karthik.
>>
>> What do you mean by parallelized WAL sender? Do you have many replicas
>> connecting to the primary? Have you tried cascading replication?
>>
>> What’s the network between the systems? Is a WAN with high latency? Can
>> the replica pull from the WAL file from an archive? For a high latency
>> networks like New York to London; I would terminate the WAL receiver via a
>> script if replication exceeded an acceptable delay to ensure SLA where
>> met. It is faster to pull the WAL files from the local archive which where
>> already replicated to London than streaming replication over a high latency
>> WAN. After It catches up, it will reconnect to streaming replication.
>>
>> --
> Thanks & Regards
> Karthik Yellapragada
> +1 860 830 5235
>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Parallelize WAL Sender and WAL receiver
2024-09-05 01:03 Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
2024-09-05 01:52 ` Re: Parallelize WAL Sender and WAL receiver Rui DeSousa <[email protected]>
2024-09-05 03:47 ` Re: Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
2024-09-05 07:54 ` Re: Parallelize WAL Sender and WAL receiver ajit wangkhem <[email protected]>
@ 2024-09-05 12:08 ` Scott Ribe <[email protected]>
0 siblings, 0 replies; 10+ messages in thread
From: Scott Ribe @ 2024-09-05 12:08 UTC (permalink / raw)
To: ajit wangkhem <[email protected]>; +Cc: Pgsql-admin <[email protected]>
> On Sep 5, 2024, at 1:54 AM, ajit wangkhem <[email protected]> wrote:
>
> If speed create problem than make 3 easy script. 1. compress wal_archive. 2. ship wal archive over network and than 3. in the secondary server just decompress it.
I'm guessing you've never benchmarked your compression? With 10GbE, the CPU overhead of compression will just slow it down.
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Parallelize WAL Sender and WAL receiver
2024-09-05 01:03 Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
@ 2024-09-06 03:33 ` Muhammad Usman Khan <[email protected]>
2024-09-06 05:02 ` Re: Parallelize WAL Sender and WAL receiver Laurenz Albe <[email protected]>
3 siblings, 1 reply; 10+ messages in thread
From: Muhammad Usman Khan @ 2024-09-06 03:33 UTC (permalink / raw)
To: Karthik Yellapragada <[email protected]>; +Cc: Pgsql-admin <[email protected]>
Hi,
You can try to set the following parameters in the conf file which may help
to overcome this problem.
wal_sender_timeout = 60s
wal_receiver_status_interval = 1s
Ensure you have sufficient WAL senders like max_wal_senders = 10
max_wal_size = 2GB
min_wal_size = 80MB
wal_buffers = 16MB
wal_level = replica
On Thu, 5 Sept 2024 at 06:04, Karthik Yellapragada <
[email protected]> wrote:
> Hello all,
>
> I frequently face the problem of wals generated faster than the wals
> transferred and applied.
>
> Is there a way to speed up the process? So I don’t accumulate a lot of
> WALs at the primary?
>
> Regards,
> Karthik.
> --
> Thanks & Regards
> Karthik Yellapragada
> +1 860 830 5235
>
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Parallelize WAL Sender and WAL receiver
2024-09-05 01:03 Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
2024-09-06 03:33 ` Re: Parallelize WAL Sender and WAL receiver Muhammad Usman Khan <[email protected]>
@ 2024-09-06 05:02 ` Laurenz Albe <[email protected]>
0 siblings, 0 replies; 10+ messages in thread
From: Laurenz Albe @ 2024-09-06 05:02 UTC (permalink / raw)
To: Muhammad Usman Khan <[email protected]>; Karthik Yellapragada <[email protected]>; +Cc: Pgsql-admin <[email protected]>
On Fri, 2024-09-06 at 08:33 +0500, Muhammad Usman Khan wrote:
> On Thu, 5 Sept 2024 at 06:04, Karthik Yellapragada <[email protected]> wrote:
> > I frequently face the problem of wals generated faster than the wals transferred and applied.
> >
> > Is there a way to speed up the process? So I don’t accumulate a lot of WALs at the primary?
> >
>
> You can try to set the following parameters in the conf file which may help to overcome this problem.
>
> wal_sender_timeout = 60s
> wal_receiver_status_interval = 1s
> Ensure you have sufficient WAL senders like max_wal_senders = 10
> max_wal_size = 2GB
> min_wal_size = 80MB
> wal_buffers = 16MB
> wal_level = replica
These settings won't do anything to mitigate the problem.
All you can do is get more network bandwidth.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2024-09-06 05:02 UTC | newest]
Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-09-05 01:03 Parallelize WAL Sender and WAL receiver Karthik Yellapragada <[email protected]>
2024-09-05 01:34 ` Ron Johnson <[email protected]>
2024-09-05 01:41 ` Keith <[email protected]>
2024-09-05 01:52 ` Rui DeSousa <[email protected]>
2024-09-05 03:47 ` Karthik Yellapragada <[email protected]>
2024-09-05 04:33 ` Rui DeSousa <[email protected]>
2024-09-05 07:54 ` ajit wangkhem <[email protected]>
2024-09-05 12:08 ` Scott Ribe <[email protected]>
2024-09-06 03:33 ` Muhammad Usman Khan <[email protected]>
2024-09-06 05:02 ` Laurenz Albe <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox