pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedWAL restore is very slow
3+ messages / 3 participants
[nested] [flat]
* WAL restore is very slow
@ 2022-08-29 10:10 Madhu Sudan <madhusudan0429@gmail.com>
2022-08-29 10:28 ` Re: WAL restore is very slow John Scalia <jayknowsunix@gmail.com>
2022-08-29 20:38 ` Re: WAL restore is very slow Jeff Janes <jeff.janes@gmail.com>
0 siblings, 2 replies; 3+ messages in thread
From: Madhu Sudan @ 2022-08-29 10:10 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hi
We have PG-14 with a huge data set of 14 TB running on r5b.2xlarge. We have
set up WAL archiving and restoring them onto a replica server. The WAL
restore on the replica is very slow and we are not able to achieve the 4
hour delayed replica. It is always behind 30 hrs with the huge WAL
generation.
I have checked the following and they look fine
1. Bottlenecks on the replica server
2. Memory consumption and swap
3. EFS IO throughput
4.checkpoint_completion_target = 0.9
5. wal_buffers = 16MB
6. wal_log_hints = on
7. Verified logs and didn't find anything useful related to the issue
Can you please suggest how to improve the WAL restore performance
Thank you
Madhu Sudan
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: WAL restore is very slow
2022-08-29 10:10 WAL restore is very slow Madhu Sudan <madhusudan0429@gmail.com>
@ 2022-08-29 10:28 ` John Scalia <jayknowsunix@gmail.com>
1 sibling, 0 replies; 3+ messages in thread
From: John Scalia @ 2022-08-29 10:28 UTC (permalink / raw)
To: Madhu Sudan <madhusudan0429@gmail.com>; +Cc: pgsql-admin@lists.postgresql.org
First off, why are you still running on a rather small r5 instance? AWS has had r6 instances available for some time, and they’re faster and more efficient. Of my some 30 instances which I take care of, I haven’t had any r5’s in quite some time.
And it sounds here that you’re running on an EC2 instance as well. Is there some reason you haven’t gone to RDS? Cluster configuration in RDS uses an internal RDS specific replication method that does not involve WAL files.
—
John
Sent from my iPad
> On Aug 29, 2022, at 6:10 AM, Madhu Sudan <madhusudan0429@gmail.com> wrote:
>
>
> Hi
>
> We have PG-14 with a huge data set of 14 TB running on r5b.2xlarge. We have set up WAL archiving and restoring them onto a replica server. The WAL restore on the replica is very slow and we are not able to achieve the 4 hour delayed replica. It is always behind 30 hrs with the huge WAL generation.
>
> I have checked the following and they look fine
> 1. Bottlenecks on the replica server
> 2. Memory consumption and swap
> 3. EFS IO throughput
> 4.checkpoint_completion_target = 0.9
> 5. wal_buffers = 16MB
> 6. wal_log_hints = on
> 7. Verified logs and didn't find anything useful related to the issue
>
> Can you please suggest how to improve the WAL restore performance
>
> Thank you
> Madhu Sudan
>
>
>
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: WAL restore is very slow
2022-08-29 10:10 WAL restore is very slow Madhu Sudan <madhusudan0429@gmail.com>
@ 2022-08-29 20:38 ` Jeff Janes <jeff.janes@gmail.com>
1 sibling, 0 replies; 3+ messages in thread
From: Jeff Janes @ 2022-08-29 20:38 UTC (permalink / raw)
To: Madhu Sudan <madhusudan0429@gmail.com>; +Cc: pgsql-admin@lists.postgresql.org
On Mon, Aug 29, 2022 at 6:10 AM Madhu Sudan <madhusudan0429@gmail.com>
wrote:
> Hi
>
> We have PG-14 with a huge data set of 14 TB running on r5b.2xlarge. We
> have set up WAL archiving and restoring them onto a replica server. The WAL
> restore on the replica is very slow and we are not able to achieve the 4
> hour delayed replica. It is always behind 30 hrs with the huge WAL
> generation.
>
> I have checked the following and they look fine
> 1. Bottlenecks on the replica server
>
How did you check for bottlenecks, and what did you see to conclude it
looked fine? Clearly there is a bottleneck somewhere.
Cheers,
Jeff
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2022-08-29 20:38 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-08-29 10:10 WAL restore is very slow Madhu Sudan <madhusudan0429@gmail.com>
2022-08-29 10:28 ` John Scalia <jayknowsunix@gmail.com>
2022-08-29 20:38 ` Jeff Janes <jeff.janes@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox