public inbox for [email protected]
help / color / mirror / Atom feedRe: Replication slot WAL reservation
2+ messages / 1 participants
[nested] [flat]
* Re: Replication slot WAL reservation
@ 2025-03-25 17:13 Christophe Pettus <[email protected]>
0 siblings, 1 reply; 2+ messages in thread
From: Christophe Pettus @ 2025-03-25 17:13 UTC (permalink / raw)
To: Phillip Diffley <[email protected]>; +Cc: pgsql-general
Missed this question!
> On Mar 25, 2025, at 09:56, Phillip Diffley <[email protected]> wrote:
> But when processing data from a replication slot, we confirm rows that have been processed and can be deleted from the WAL based on the LSN (eg. with pg_replication_slot_advance). How does postgres identify what parts of the WAL can be freed?
Basically, if no part of the system "needs" a particular LSN position, the segments that include that LSN position and earlier can be free.
The various things that can "need" a particular LSN point are:
1. Replication slots, if the other side has not confirmed that it has received it (under whatever synchronous commit rules that slot is operating under).
2. The wal_keep_size setting.
3. The max_wal_size setting.
4. The archive_command, if a WAL segment hasn't been successfully archived yet.
One thing to remember is that the WAL does *not* contain contiguous blocks of operations for a single transaction. The operations are written to the WAL by every session as they do operations, so the WAL is a jumble of different transactions. One of the jobs of the logical replication framework is to sort that out so it can present only the operations that belong to committed transactions to the output plugin. (This is why there's an internal structure called the "reorder buffer": it reorders WAL operations into transaction blocks.)
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: Replication slot WAL reservation
@ 2025-03-25 21:31 Christophe Pettus <[email protected]>
parent: Christophe Pettus <[email protected]>
0 siblings, 0 replies; 2+ messages in thread
From: Christophe Pettus @ 2025-03-25 21:31 UTC (permalink / raw)
To: Phillip Diffley <[email protected]>; +Cc: pgsql-general
> On Mar 25, 2025, at 13:58, Phillip Diffley <[email protected]> wrote:
>
> Oh I see! I was conflating the data I see coming out of a replication slot with the internal organization of the WAL. I think the more specific question I am trying to answer is, as a consumer of a replication slot, how do I reason about what replication records will be made unavailable when I confirm an LSN? Here I am worried about situations where the replication connection is interrupted or the program processing the records crashes, and we need to replay records that may have been previously sent but were not fully processed.
It's up to the consuming client to keep track of where it is in the WAL (using an LSN). When the client connects, it specifies what LSN to start streaming at. If that LSN is no longer available, the publisher / primary returns an error.
The client shouldn't confirm the flush of an LSN unless it is crash-proof to that point, since any WAL before that should be assumed to be unavailable.
> For example, are the records sent by a replication slot always sent in the same order such that if I advance the confirmed_flush_lsn of a slot to the LSN of record "A", I will know that any records that had been streamed after record "A" will be replayable?
You know that any WAL generated after `confirmed_flush_lsn` is available for replay. That's the oldest LSN that the client can specify on connection (although it can specify a later one, if it exists). You shouldn't need to manually advance the replication slot. Instead, the client specifies where it wants to start when it connects. The client is also expected to send back regular messages letting the publisher / primary know that it has successfully consumed up to a particular point in the WAL, so the publisher / primary knows it can release that WAL information.
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2025-03-25 21:31 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-03-25 17:13 Re: Replication slot WAL reservation Christophe Pettus <[email protected]>
2025-03-25 21:31 ` Christophe Pettus <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox