Re: RDS restore failed due to WAL log and disk space-- any tidy fixes?

public inbox for [email protected]  
help / color / mirror / Atom feed

From: Wells Oliver <[email protected]>
To: Ron Johnson <[email protected]>
Cc: pgsql-admin <[email protected]>
Subject: Re: RDS restore failed due to WAL log and disk space-- any tidy fixes?
Date: Sun, 17 Nov 2024 09:31:19 -0800
Message-ID: <CAOC+FBVHX5yPrawa6wQh6rC4F6GJYMHK9zjH-DhF7xV-g0PaZQ@mail.gmail.com> (raw)
In-Reply-To: <CAOC+FBUoykok2udm_3S8U4wRTDbKGdf=oY0Ga6QDCbLZ9L4wnw@mail.gmail.com>
References: <CAOC+FBVyTiuOMF92pQG3QOrE9j8qFegvZLoi-3UmdhFNg94e2A@mail.gmail.com>
	<[email protected]>
	<CAOC+FBX1L6hx4DHzH34eaWyhH6NK9ywYSJMMcEJH3rHZfW86+A@mail.gmail.com>
	<CANzqJaAh+p4Hys-aPTW2zAuYg_MLj7QJ7J6tQB3g8dA2K3DtWg@mail.gmail.com>
	<CAOC+FBUoykok2udm_3S8U4wRTDbKGdf=oY0Ga6QDCbLZ9L4wnw@mail.gmail.com>

Actually, in RDS it seems you cannot set archive_mode either.

On Sun, Nov 17, 2024 at 9:23 AM Wells Oliver <[email protected]> wrote:

> It does. I think it uses WAL behind the scenes. In RDS unfortunately
> cannot set wal_level, but you can set archive_mode.
>
> On Sun, Nov 17, 2024 at 9:21 AM Ron Johnson <[email protected]>
> wrote:
>
>> Doesn't RDS have its own replication?
>>
>> Anyway, for pg_restore, I'd absolutely set archive_mode=off
>> and wal_level=minimal, then set them to their production values when it's
>> finished.
>>
>> On Sun, Nov 17, 2024 at 12:12 PM Wells Oliver <[email protected]>
>> wrote:
>>
>>> Interesting. I am migrating a pg_dump archive to a new server, in a
>>> single go. Does it make sense to disable (or speed up?) WAL archiving
>>> during the restore, then reenable it after the restore so a future replica
>>> could work? What would be the steps here? Would disabling or "speeding up"
>>> be faster?
>>>
>>> max_slot_wal_keep_size is -1 at the moment so I think that's why it kept
>>> a ton of WAL and ran out of space.
>>>
>>> On Sun, Nov 17, 2024 at 7:41 AM Laurenz Albe <[email protected]>
>>> wrote:
>>>
>>>> On Sat, 2024-11-16 at 16:33 -0800, Wells Oliver wrote:
>>>> > I provisioned an RDS instance with 2500GB space and began the restore
>>>> of a database I know to be about 1750 GB using 16 jobs.
>>>> >
>>>> > Unfortunately, it died very near the end when it ran out of disk
>>>> space due to WAL log usage. Lots of:
>>>> >
>>>> > 2024-11-17 00:07:09 UTC::@:[19861]:PANIC:  could not write to file
>>>> "pg_wal/xlogtemp.19861": No space left on device
>>>> >
>>>> >
>>>> > And then kaboom.
>>>> >
>>>> > I'm wondering what my course of action should be. Can I
>>>> disable/reduce WAL during a restore?
>>>> > wal_level is set to replica, can this temporarily be set to minimal?
>>>> Should I just eat the extra
>>>> > costs to add headroom for the WAL? Would using fewer jobs during a
>>>> restore reduce the amount of WAL
>>>> > created?
>>>>
>>>> If you are using minimal WAL logging and you restore the dump in a
>>>> single transaction, you
>>>> should see way less WAL generated, because data inserted into the table
>>>> in the same transaction
>>>> as the CREATE TABLE statement need not be WAL logged.
>>>>
>>>> But you might more easily solve the problem by speeding up or disabling
>>>> the WAL archiver,
>>>> so that PostgreSQL removes old WAL after the next checkpoint.
>>>>
>>>> Yours,
>>>> Laurenz Albe
>>>>
>>>
>>>
>>> --
>>> Wells Oliver
>>> [email protected] <[email protected]>
>>>
>>
>>
>> --
>> Death to <Redacted>, and butter sauce.
>> Don't boil me, I'm still alive.
>> <Redacted> lobster!
>>
>
>
> --
> Wells Oliver
> [email protected] <[email protected]>
>


-- 
Wells Oliver
[email protected] <[email protected]>

view thread (8+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected]
  Subject: Re: RDS restore failed due to WAL log and disk space-- any tidy fixes?
  In-Reply-To: <CAOC+FBVHX5yPrawa6wQh6rC4F6GJYMHK9zjH-DhF7xV-g0PaZQ@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox