public inbox for [email protected]
help / color / mirror / Atom feedRe: Questions about the continuity of WAL archiving
5+ messages / 3 participants
[nested] [flat]
* Re: Questions about the continuity of WAL archiving
@ 2025-08-08 18:25 Greg Sabino Mullane <[email protected]>
2025-08-08 18:45 ` Re: Questions about the continuity of WAL archiving Ron Johnson <[email protected]>
2025-08-12 08:33 ` Re: Questions about the continuity of WAL archiving px shi <[email protected]>
0 siblings, 2 replies; 5+ messages in thread
From: Greg Sabino Mullane @ 2025-08-08 18:25 UTC (permalink / raw)
To: px shi <[email protected]>; +Cc: [email protected]
>
> There is a scenario: the current timeline of the PostgreSQL primary node
> is 1, and the latest WAL file is 100. The standby node has also received up
> to WAL file 100. However, the latest WAL file archived is only file 80. If
> the primary node crashes at this point and the standby is promoted to the
> new primary, archiving will resume from file 100 on timeline 2. As a
> result, WAL files from 81 to 100 on timeline 1 will be missing from the
> archive.
> Is there a good solution to prevent this situation?
>
I'm still not clear on what the problem here is, other than your archiving
not keeping up. The best solution to that is:
https://pgbackrest.org/1/configuration.html#section-archive/option-archive-async
Yes, you would lost some ability for easy PITR for 80-100, but could still
be done by resurrecting your crashed primary, or carefully grabbing from
the replica before they get recycled. You can set archive_mode=always on
the replicas to help with this.
Cheers,
Greg
--
Crunchy Data - https://www.crunchydata.com
Enterprise Postgres Software Products & Tech Support
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Questions about the continuity of WAL archiving
2025-08-08 18:25 Re: Questions about the continuity of WAL archiving Greg Sabino Mullane <[email protected]>
@ 2025-08-08 18:45 ` Ron Johnson <[email protected]>
2025-08-12 08:37 ` Re: Questions about the continuity of WAL archiving px shi <[email protected]>
1 sibling, 1 reply; 5+ messages in thread
From: Ron Johnson @ 2025-08-08 18:45 UTC (permalink / raw)
To: pgsql-generallists.postgresql.org <[email protected]>
On Fri, Aug 8, 2025 at 2:26 PM Greg Sabino Mullane <[email protected]>
wrote:
> There is a scenario: the current timeline of the PostgreSQL primary node
>> is 1, and the latest WAL file is 100. The standby node has also received up
>> to WAL file 100. However, the latest WAL file archived is only file 80. If
>> the primary node crashes at this point and the standby is promoted to the
>> new primary, archiving will resume from file 100 on timeline 2. As a
>> result, WAL files from 81 to 100 on timeline 1 will be missing from the
>> archive.
>> Is there a good solution to prevent this situation?
>>
>
> I'm still not clear on what the problem here is, other than your archiving
> not keeping up. The best solution to that is:
>
>
> https://pgbackrest.org/1/configuration.html#section-archive/option-archive-async
>
> Yes, you would lost some ability for easy PITR for 80-100, but could still
> be done by resurrecting your crashed primary, or carefully grabbing from
> the replica before they get recycled. You can set archive_mode=always on
> the replicas to help with this.
>
Bog-standard PgBackRest retains all WAL files required for a full backup
set and its associated differential/incremental backups, no? I've
certainly done more than one --type=time --target="${RestoreUntil}" restore
without giving a second thought to timelines or whether the WAL exists.
Maybe I've just ignored the problem, since it (seemingly) does everything
for PITR backups.
--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Questions about the continuity of WAL archiving
2025-08-08 18:25 Re: Questions about the continuity of WAL archiving Greg Sabino Mullane <[email protected]>
2025-08-08 18:45 ` Re: Questions about the continuity of WAL archiving Ron Johnson <[email protected]>
@ 2025-08-12 08:37 ` px shi <[email protected]>
2025-08-12 17:51 ` Re: Questions about the continuity of WAL archiving Ron Johnson <[email protected]>
0 siblings, 1 reply; 5+ messages in thread
From: px shi @ 2025-08-12 08:37 UTC (permalink / raw)
To: Ron Johnson <[email protected]>; +Cc: pgsql-generallists.postgresql.org <[email protected]>
>
> Bog-standard PgBackRest retains all WAL files required for a full backup
> set and its associated differential/incremental backups.
Yes, WAL files are continuous under normal circumstances. However, if the
primary node crashes under high load, the archived WAL logs on S3 may be
discontinuous.
Ron Johnson <[email protected]> 于2025年8月9日周六 02:45写道：
> On Fri, Aug 8, 2025 at 2:26 PM Greg Sabino Mullane <[email protected]>
> wrote:
>
>> There is a scenario: the current timeline of the PostgreSQL primary node
>>> is 1, and the latest WAL file is 100. The standby node has also received up
>>> to WAL file 100. However, the latest WAL file archived is only file 80. If
>>> the primary node crashes at this point and the standby is promoted to the
>>> new primary, archiving will resume from file 100 on timeline 2. As a
>>> result, WAL files from 81 to 100 on timeline 1 will be missing from the
>>> archive.
>>> Is there a good solution to prevent this situation?
>>>
>>
>> I'm still not clear on what the problem here is, other than your
>> archiving not keeping up. The best solution to that is:
>>
>>
>> https://pgbackrest.org/1/configuration.html#section-archive/option-archive-async
>>
>> Yes, you would lost some ability for easy PITR for 80-100, but could
>> still be done by resurrecting your crashed primary, or carefully grabbing
>> from the replica before they get recycled. You can set archive_mode=always
>> on the replicas to help with this.
>>
>
> Bog-standard PgBackRest retains all WAL files required for a full backup
> set and its associated differential/incremental backups, no? I've
> certainly done more than one --type=time --target="${RestoreUntil}" restore
> without giving a second thought to timelines or whether the WAL exists.
>
> Maybe I've just ignored the problem, since it (seemingly) does everything
> for PITR backups.
>
> --
> Death to <Redacted>, and butter sauce.
> Don't boil me, I'm still alive.
> <Redacted> lobster!
>
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Questions about the continuity of WAL archiving
2025-08-08 18:25 Re: Questions about the continuity of WAL archiving Greg Sabino Mullane <[email protected]>
2025-08-08 18:45 ` Re: Questions about the continuity of WAL archiving Ron Johnson <[email protected]>
2025-08-12 08:37 ` Re: Questions about the continuity of WAL archiving px shi <[email protected]>
@ 2025-08-12 17:51 ` Ron Johnson <[email protected]>
0 siblings, 0 replies; 5+ messages in thread
From: Ron Johnson @ 2025-08-12 17:51 UTC (permalink / raw)
To: pgsql-general
On Tue, Aug 12, 2025 at 4:37 AM px shi <[email protected]> wrote:
> Bog-standard PgBackRest retains all WAL files required for a full backup
>> set and its associated differential/incremental backups.
>
> Yes, WAL files are continuous under normal circumstances. However, if the
> primary node crashes under high load, the archived WAL logs on S3 may be
> discontinuous.
>
1) PG does not purge WAL files that are needed for immediate crash recovery.
2) PgBackRest can archive (compressed and encrypted) WAL files to S3.
https://pgbackrest.org/user-guide-rhel.html#s3-support
>
> Ron Johnson <[email protected]> 于2025年8月9日周六 02:45写道：
>
>> On Fri, Aug 8, 2025 at 2:26 PM Greg Sabino Mullane <[email protected]>
>> wrote:
>>
>>> There is a scenario: the current timeline of the PostgreSQL primary node
>>>> is 1, and the latest WAL file is 100. The standby node has also received up
>>>> to WAL file 100. However, the latest WAL file archived is only file 80. If
>>>> the primary node crashes at this point and the standby is promoted to the
>>>> new primary, archiving will resume from file 100 on timeline 2. As a
>>>> result, WAL files from 81 to 100 on timeline 1 will be missing from the
>>>> archive.
>>>> Is there a good solution to prevent this situation?
>>>>
>>>
>>> I'm still not clear on what the problem here is, other than your
>>> archiving not keeping up. The best solution to that is:
>>>
>>>
>>> https://pgbackrest.org/1/configuration.html#section-archive/option-archive-async
>>>
>>> Yes, you would lost some ability for easy PITR for 80-100, but could
>>> still be done by resurrecting your crashed primary, or carefully grabbing
>>> from the replica before they get recycled. You can set archive_mode=always
>>> on the replicas to help with this.
>>>
>>
>> Bog-standard PgBackRest retains all WAL files required for a full backup
>> set and its associated differential/incremental backups, no? I've
>> certainly done more than one --type=time --target="${RestoreUntil}" restore
>> without giving a second thought to timelines or whether the WAL exists.
>>
>> Maybe I've just ignored the problem, since it (seemingly) does everything
>> for PITR backups.
>>
>
--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Questions about the continuity of WAL archiving
2025-08-08 18:25 Re: Questions about the continuity of WAL archiving Greg Sabino Mullane <[email protected]>
@ 2025-08-12 08:33 ` px shi <[email protected]>
1 sibling, 0 replies; 5+ messages in thread
From: px shi @ 2025-08-12 08:33 UTC (permalink / raw)
To: Greg Sabino Mullane <[email protected]>; +Cc: [email protected]
>
> I'm still not clear on what the problem here is, other than your archiving
> not keeping up.
In my scenario, archive_mode is not set to always on the replicas, it may
cause interruptions in the archived WAL logs.
You can set archive_mode=always on the replicas to help with this.
Yes, it can work. And I would like to know if this is the recommended
configuration for production use?
Greg Sabino Mullane <[email protected]> 于2025年8月9日周六 02:25写道：
> There is a scenario: the current timeline of the PostgreSQL primary node
>> is 1, and the latest WAL file is 100. The standby node has also received up
>> to WAL file 100. However, the latest WAL file archived is only file 80. If
>> the primary node crashes at this point and the standby is promoted to the
>> new primary, archiving will resume from file 100 on timeline 2. As a
>> result, WAL files from 81 to 100 on timeline 1 will be missing from the
>> archive.
>> Is there a good solution to prevent this situation?
>>
>
> I'm still not clear on what the problem here is, other than your archiving
> not keeping up. The best solution to that is:
>
>
> https://pgbackrest.org/1/configuration.html#section-archive/option-archive-async
>
> Yes, you would lost some ability for easy PITR for 80-100, but could still
> be done by resurrecting your crashed primary, or carefully grabbing from
> the replica before they get recycled. You can set archive_mode=always on
> the replicas to help with this.
>
> Cheers,
> Greg
>
> --
> Crunchy Data - https://www.crunchydata.com
> Enterprise Postgres Software Products & Tech Support
>
>
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2025-08-12 17:51 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-08-08 18:25 Re: Questions about the continuity of WAL archiving Greg Sabino Mullane <[email protected]>
2025-08-08 18:45 ` Ron Johnson <[email protected]>
2025-08-12 08:37 ` px shi <[email protected]>
2025-08-12 17:51 ` Ron Johnson <[email protected]>
2025-08-12 08:33 ` px shi <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox