public inbox for [email protected]
help / color / mirror / Atom feedFrom: Ron Johnson <[email protected]>
To: pgsql-admin <[email protected]>
Subject: Re: RDS restore failed due to WAL log and disk space-- any tidy fixes?
Date: Sun, 17 Nov 2024 12:21:05 -0500
Message-ID: <CANzqJaAh+p4Hys-aPTW2zAuYg_MLj7QJ7J6tQB3g8dA2K3DtWg@mail.gmail.com> (raw)
In-Reply-To: <CAOC+FBX1L6hx4DHzH34eaWyhH6NK9ywYSJMMcEJH3rHZfW86+A@mail.gmail.com>
References: <CAOC+FBVyTiuOMF92pQG3QOrE9j8qFegvZLoi-3UmdhFNg94e2A@mail.gmail.com>
<[email protected]>
<CAOC+FBX1L6hx4DHzH34eaWyhH6NK9ywYSJMMcEJH3rHZfW86+A@mail.gmail.com>
Doesn't RDS have its own replication?
Anyway, for pg_restore, I'd absolutely set archive_mode=off
and wal_level=minimal, then set them to their production values when it's
finished.
On Sun, Nov 17, 2024 at 12:12 PM Wells Oliver <[email protected]>
wrote:
> Interesting. I am migrating a pg_dump archive to a new server, in a single
> go. Does it make sense to disable (or speed up?) WAL archiving during the
> restore, then reenable it after the restore so a future replica could work?
> What would be the steps here? Would disabling or "speeding up" be faster?
>
> max_slot_wal_keep_size is -1 at the moment so I think that's why it kept a
> ton of WAL and ran out of space.
>
> On Sun, Nov 17, 2024 at 7:41 AM Laurenz Albe <[email protected]>
> wrote:
>
>> On Sat, 2024-11-16 at 16:33 -0800, Wells Oliver wrote:
>> > I provisioned an RDS instance with 2500GB space and began the restore
>> of a database I know to be about 1750 GB using 16 jobs.
>> >
>> > Unfortunately, it died very near the end when it ran out of disk space
>> due to WAL log usage. Lots of:
>> >
>> > 2024-11-17 00:07:09 UTC::@:[19861]:PANIC: could not write to file
>> "pg_wal/xlogtemp.19861": No space left on device
>> >
>> >
>> > And then kaboom.
>> >
>> > I'm wondering what my course of action should be. Can I disable/reduce
>> WAL during a restore?
>> > wal_level is set to replica, can this temporarily be set to minimal?
>> Should I just eat the extra
>> > costs to add headroom for the WAL? Would using fewer jobs during a
>> restore reduce the amount of WAL
>> > created?
>>
>> If you are using minimal WAL logging and you restore the dump in a single
>> transaction, you
>> should see way less WAL generated, because data inserted into the table
>> in the same transaction
>> as the CREATE TABLE statement need not be WAL logged.
>>
>> But you might more easily solve the problem by speeding up or disabling
>> the WAL archiver,
>> so that PostgreSQL removes old WAL after the next checkpoint.
>>
>> Yours,
>> Laurenz Albe
>>
>
>
> --
> Wells Oliver
> [email protected] <[email protected]>
>
--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!
view thread (8+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected]
Subject: Re: RDS restore failed due to WAL log and disk space-- any tidy fixes?
In-Reply-To: <CANzqJaAh+p4Hys-aPTW2zAuYg_MLj7QJ7J6tQB3g8dA2K3DtWg@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox