From: MichaelDBA <MichaelDBA@sqlexec.com>
To: Nikhil Shetty <nikhil.dba04@gmail.com>
Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Subject: Re: Parallel WAL Archival Options
Date: Sun, 6 Aug 2023 08:55:10 -0400
Message-ID: <12e141e5-8d87-349b-7c9a-88dd7974eee9@sqlexec.com> (raw)
In-Reply-To: <CAFpL5Vz0oDBgNP5UTShf6y8MzMm4z0Di0ZBFFVYR67QaJ_YObA@mail.gmail.com>
References: <CAFpL5Vx86qhPAiOsUyrK=7hwKTuufbYYPp6kEnDJwvb=ZRphYg@mail.gmail.com>
<2A210A32-AB11-4587-9EB6-C0D8626D83E9@sqlexec.com>
<CAFpL5Vz0oDBgNP5UTShf6y8MzMm4z0Di0ZBFFVYR67QaJ_YObA@mail.gmail.com>
I do about 2-10 WALs per second. I have a remote pgbackrest repo, and I
do faster WAL archiving using the archive-async=y parameter, which
enables WALs to be archived asynchronously, really speeding it up. I
also increase the archive-push-queue-max=100GB.
For fetching the WALs, like when a standby gets behind and it needs to
read WALs to catch up to get back into SR mode, I increase the size of
the pgBackRest queue on the standby ( archive_get-queue-max = 100GB).
Both queue max parms are only enabled when archive-async is turned on.
Regards,
Michael Vitale
Nikhil Shetty wrote on 8/6/2023 6:56 AM:
> Hi Michael,
>
> May I know what is wal generation rate pm for you? I may give this a try.
>
>
> Thanks,
> Nikhil
>
> On Sun, 6 Aug 2023 at 13:56, Michaeldba@sqlexec.com
> <mailto:Michaeldba@sqlexec.com> <Michaeldba@sqlexec.com
> <mailto:Michaeldba@sqlexec.com>> wrote:
>
> I am using PGbackrest and I am generating many WALs per second
> too, and I have no problems keeping up and I am backing them up
> remotely.
>
> Sent from my iPad
>
> > On Aug 6, 2023, at 3:43 AM, Nikhil Shetty
> <nikhil.dba04@gmail.com <mailto:nikhil.dba04@gmail.com>> wrote:
> >
> > Hi Team,
> >
> > I would like to know which backup&restore tools will be better
> for scenarios where the database is generating around 400 WALs per
> minute.
> >
> > We are using wal-g but it is not able to keep pace with the wal
> generation. We increased the upload streams to 256 but no luck
> >
> > Thanks,
> > Nikhil
>
Regards,
Michael Vitale
Michaeldba@sqlexec.com <mailto:michaelvitale@sqlexec.com>
703-600-9343
Attachments:
[image/jpeg] pgadvanced3.jpg (20.6K, ../12e141e5-8d87-349b-7c9a-88dd7974eee9@sqlexec.com/3-pgadvanced3.jpg)
download | view image
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: MichaelDBA@sqlexec.com, nikhil.dba04@gmail.com, pgsql-admin@lists.postgresql.org
Subject: Re: Parallel WAL Archival Options
In-Reply-To: <12e141e5-8d87-349b-7c9a-88dd7974eee9@sqlexec.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox