public inbox for [email protected]
help / color / mirror / Atom feedFrom: Shubhang Joshi <[email protected]>
To: OMPRAKASH SAHU <[email protected]>
Cc: [email protected]
Subject: Re: WAL replay is too slow on secondary server
Date: Thu, 30 Oct 2025 17:08:06 +0530
Message-ID: <CAOJCrX91Xf3HU5J0Vn_FdrRDpMevNiZUEN3oAWwk4J1H0ibo-Q@mail.gmail.com> (raw)
In-Reply-To: <CAOZWJqPc+s_vA-UfWWLR0s6Mt+DCffjXXVyLHJNJiuMrDLTYcA@mail.gmail.com>
References: <CAOZWJqPc+s_vA-UfWWLR0s6Mt+DCffjXXVyLHJNJiuMrDLTYcA@mail.gmail.com>
Hi OM,
Please check the network speed — we faced a similar issue earlier, and it
turned out to be related to network performance.
Kindly verify the network latency with your network team as well.
Regards,
Shubhang
On Thu, 30 Oct, 2025, 10:07 am OMPRAKASH SAHU, <[email protected]> wrote:
> Hi Team,
>
> Greetings!!
>
> We have a postgresql cluster setup using patroni.
> The DB is being used for heavy transactional application, now the
> problem is that on replica server WAL replay is too slow.
> We have increased the IOPS to 6k and Throughput to 600 on nvme EBS volume
> of wal directory and 10k &800 on data directory.
>
> but the WAL is being accumulated on the replica as usual and applying wal
> is having no improvement.
> changed the maintenance_io_concurrency on replica to 32.
> CPU utilization max=20% , RAM utilization is also max 20.
>
> see the below postgres logs that shows around 2hrs lag
>
>
>
>
>
>
>
> *tail -f /var/log/postgresql/postgresql.log2025-10-30 09:02:08 IST
> [27125]: user=,db=,app=,client=LOG: recovery restart point at
> 5B65/F1DAFA202025-10-30 09:02:08 IST [27125]: user=,db=,app=,client=DETAIL:
> Last completed transaction was at log time 2025-10-30
> 07:16:40.115131+05:30.2025-10-30 09:08:23 IST [27125]:
> user=,db=,app=,client=LOG: restartpoint starting: time2025-10-30 09:12:53
> IST [27125]: user=,db=,app=,client=LOG: restartpoint complete: wrote 44067
> buffers (2.1%); 1 WAL file(s) added, 73 removed, 0 recycled; write=269.362
> s, sync=0.042 s, total=269.633 s; sync files=142, longest=0.005 s,
> average=0.001 s; distance=1197052 kB, estimate=1587336 kB;
> lsn=5B66/6C3082F8, redo lsn=5B66/3AEAEA602025-10-30 09:12:53 IST [27125]:
> user=,db=,app=,client=LOG: recovery restart point at
> 5B66/3AEAEA602025-10-30 09:12:53 IST [27125]: user=,db=,app=,client=DETAIL:
> Last completed transaction was at log time 2025-10-30
> 07:21:47.56674+05:30.*
>
> recovery_prefetch output:
>
> postgres=# select * from pg_stat_recovery_prefetch;
> stats_reset | prefetch | hit | skip_init |
> skip_new | skip_fpw | skip_rep | wal_distance | block_distance | io_depth
>
> ----------------------------------+----------+-----------+-----------+----------+----------+-----------+--------------+----------------+----------
> 2025-10-29 23:02:21.396179+05:30 | 182762 | 251000856 | 3841721 |
> 1100099 | 3520777 | 137392573 | 8984 | 80 | 0
> (1 row)
>
> I would request your thoughts and suggestions if we can get rid of this
> slowness and get some speed.
>
> Regards,
> OM
>
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: WAL replay is too slow on secondary server
In-Reply-To: <CAOJCrX91Xf3HU5J0Vn_FdrRDpMevNiZUEN3oAWwk4J1H0ibo-Q@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox