public inbox for [email protected]help / color / mirror / Atom feed
Re: Introduce XID age and inactive timeout based replication slot invalidation 46+ messages / 6 participants [nested] [flat]
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 08:07 Amit Kapila <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-03-26 08:07 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Mar 26, 2024 at 1:15 PM Bertrand Drouvot <[email protected]> wrote: > > 2 === > > It looks like inactive_since is set to the current timestamp on the standby > each time the sync worker does a cycle: > > primary: > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't'; > slot_name | inactive_since > -------------+------------------------------- > lsub27_slot | 2024-03-26 07:39:19.745517+00 > lsub28_slot | 2024-03-26 07:40:24.953826+00 > > standby: > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't'; > slot_name | inactive_since > -------------+------------------------------- > lsub27_slot | 2024-03-26 07:43:56.387324+00 > lsub28_slot | 2024-03-26 07:43:56.387338+00 > > I don't think that should be the case. > But why? This is exactly what we discussed in another thread where we agreed to update inactive_since even for sync slots. In each sync cycle, we acquire/release the slot, so the inactive_since gets updated. See synchronize_one_slot(). -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 08:24 Bertrand Drouvot <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-26 08:24 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Tue, Mar 26, 2024 at 01:37:21PM +0530, Amit Kapila wrote: > On Tue, Mar 26, 2024 at 1:15 PM Bertrand Drouvot > <[email protected]> wrote: > > > > 2 === > > > > It looks like inactive_since is set to the current timestamp on the standby > > each time the sync worker does a cycle: > > > > primary: > > > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't'; > > slot_name | inactive_since > > -------------+------------------------------- > > lsub27_slot | 2024-03-26 07:39:19.745517+00 > > lsub28_slot | 2024-03-26 07:40:24.953826+00 > > > > standby: > > > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't'; > > slot_name | inactive_since > > -------------+------------------------------- > > lsub27_slot | 2024-03-26 07:43:56.387324+00 > > lsub28_slot | 2024-03-26 07:43:56.387338+00 > > > > I don't think that should be the case. > > > > But why? This is exactly what we discussed in another thread where we > agreed to update inactive_since even for sync slots. Hum, I thought we agreed to "sync" it and to "update it to current time" only at promotion time. I don't think updating inactive_since to current time during each cycle makes sense (I mean I understand the use case: being able to say when slots have been sync, but if this is what we want then we should consider an extra view or an extra field but not relying on the inactive_since one). If the primary goes down, not updating inactive_since to the current time could also provide benefit such as knowing the inactive_since of the primary slots (from the standby) the last time it has been synced. If we update it to the current time then this information is lost. > In each sync > cycle, we acquire/release the slot, so the inactive_since gets > updated. See synchronize_one_slot(). Right, and I think we should put an extra condition if in recovery. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 09:47 shveta malik <[email protected]> parent: Bertrand Drouvot <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: shveta malik @ 2024-03-26 09:47 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bharath Rupireddy <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Tue, Mar 26, 2024 at 1:54 PM Bertrand Drouvot <[email protected]> wrote: > > Hi, > > On Tue, Mar 26, 2024 at 01:37:21PM +0530, Amit Kapila wrote: > > On Tue, Mar 26, 2024 at 1:15 PM Bertrand Drouvot > > <[email protected]> wrote: > > > > > > 2 === > > > > > > It looks like inactive_since is set to the current timestamp on the standby > > > each time the sync worker does a cycle: > > > > > > primary: > > > > > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't'; > > > slot_name | inactive_since > > > -------------+------------------------------- > > > lsub27_slot | 2024-03-26 07:39:19.745517+00 > > > lsub28_slot | 2024-03-26 07:40:24.953826+00 > > > > > > standby: > > > > > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't'; > > > slot_name | inactive_since > > > -------------+------------------------------- > > > lsub27_slot | 2024-03-26 07:43:56.387324+00 > > > lsub28_slot | 2024-03-26 07:43:56.387338+00 > > > > > > I don't think that should be the case. > > > > > > > But why? This is exactly what we discussed in another thread where we > > agreed to update inactive_since even for sync slots. > > Hum, I thought we agreed to "sync" it and to "update it to current time" > only at promotion time. I think there may have been some misunderstanding here. But now if I rethink this, I am fine with 'inactive_since' getting synced from primary to standby. But if we do that, we need to add docs stating "inactive_since" represents primary's inactivity and not standby's slots inactivity for synced slots. The reason for this clarification is that the synced slot might be generated much later, yet 'inactive_since' is synced from the primary, potentially indicating a time considerably earlier than when the synced slot was actually created. Another approach could be that "inactive_since" for synced slot actually gives its own inactivity data rather than giving primary's slot data. We update inactive_since on standby only at 3 occasions: 1) at the time of creation of the synced slot. 2) during standby restart. 3) during promotion of standby. I have attached a sample patch for this idea as.txt file. I am fine with any of these approaches. One gives data synced from primary for synced slots, while another gives actual inactivity data of synced slots. thanks Shveta From 7dcd0e95299263187eb1f03812f8321b2612ee5c Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Tue, 26 Mar 2024 14:42:25 +0530 Subject: [PATCH v1] inactive_since for synced slots. inactive_since is updated for synced slots: 1) at the time of creation of slot. 2) during server restart. 3) during promotion. --- src/backend/replication/logical/slotsync.c | 1 + src/backend/replication/slot.c | 15 ++++++++++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index bbf9a2c485..6114895dca 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -628,6 +628,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) SpinLockAcquire(&slot->mutex); slot->effective_catalog_xmin = xmin_horizon; slot->data.catalog_xmin = xmin_horizon; + slot->inactive_since = GetCurrentTimestamp(); SpinLockRelease(&slot->mutex); ReplicationSlotsComputeRequiredXmin(true); LWLockRelease(ProcArrayLock); diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index d0a2f440ef..f2a57a14ec 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -628,7 +628,10 @@ retry: * now. */ SpinLockAcquire(&s->mutex); - s->inactive_since = 0; + + if (!(RecoveryInProgress() && s->data.synced)) + s->inactive_since = 0; + SpinLockRelease(&s->mutex); if (am_walsender) @@ -704,14 +707,20 @@ ReplicationSlotRelease(void) */ SpinLockAcquire(&slot->mutex); slot->active_pid = 0; - slot->inactive_since = now; + + if (!(RecoveryInProgress() && slot->data.synced)) + slot->inactive_since = now; + SpinLockRelease(&slot->mutex); ConditionVariableBroadcast(&slot->active_cv); } else { SpinLockAcquire(&slot->mutex); - slot->inactive_since = now; + + if (!(RecoveryInProgress() && slot->data.synced)) + slot->inactive_since = now; + SpinLockRelease(&slot->mutex); } -- 2.34.1 Attachments: [text/plain] v1-0001-inactive_since-for-synced-slots.patch.txt (2.0K, ../../CAJpy0uDkTW+t1k3oPkaipFBzZePfFNB5DmiA==pxRGcAdpF=Pg@mail.gmail.com/2-v1-0001-inactive_since-for-synced-slots.patch.txt) download | inline diff: From 7dcd0e95299263187eb1f03812f8321b2612ee5c Mon Sep 17 00:00:00 2001 From: Shveta Malik <[email protected]> Date: Tue, 26 Mar 2024 14:42:25 +0530 Subject: [PATCH v1] inactive_since for synced slots. inactive_since is updated for synced slots: 1) at the time of creation of slot. 2) during server restart. 3) during promotion. --- src/backend/replication/logical/slotsync.c | 1 + src/backend/replication/slot.c | 15 ++++++++++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index bbf9a2c485..6114895dca 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -628,6 +628,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) SpinLockAcquire(&slot->mutex); slot->effective_catalog_xmin = xmin_horizon; slot->data.catalog_xmin = xmin_horizon; + slot->inactive_since = GetCurrentTimestamp(); SpinLockRelease(&slot->mutex); ReplicationSlotsComputeRequiredXmin(true); LWLockRelease(ProcArrayLock); diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index d0a2f440ef..f2a57a14ec 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -628,7 +628,10 @@ retry: * now. */ SpinLockAcquire(&s->mutex); - s->inactive_since = 0; + + if (!(RecoveryInProgress() && s->data.synced)) + s->inactive_since = 0; + SpinLockRelease(&s->mutex); if (am_walsender) @@ -704,14 +707,20 @@ ReplicationSlotRelease(void) */ SpinLockAcquire(&slot->mutex); slot->active_pid = 0; - slot->inactive_since = now; + + if (!(RecoveryInProgress() && slot->data.synced)) + slot->inactive_since = now; + SpinLockRelease(&slot->mutex); ConditionVariableBroadcast(&slot->active_cv); } else { SpinLockAcquire(&slot->mutex); - slot->inactive_since = now; + + if (!(RecoveryInProgress() && slot->data.synced)) + slot->inactive_since = now; + SpinLockRelease(&slot->mutex); } -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 10:20 Bertrand Drouvot <[email protected]> parent: shveta malik <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-26 10:20 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bharath Rupireddy <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Tue, Mar 26, 2024 at 03:17:36PM +0530, shveta malik wrote: > On Tue, Mar 26, 2024 at 1:54 PM Bertrand Drouvot > <[email protected]> wrote: > > > > Hi, > > > > On Tue, Mar 26, 2024 at 01:37:21PM +0530, Amit Kapila wrote: > > > On Tue, Mar 26, 2024 at 1:15 PM Bertrand Drouvot > > > <[email protected]> wrote: > > > > > > > > 2 === > > > > > > > > It looks like inactive_since is set to the current timestamp on the standby > > > > each time the sync worker does a cycle: > > > > > > > > primary: > > > > > > > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't'; > > > > slot_name | inactive_since > > > > -------------+------------------------------- > > > > lsub27_slot | 2024-03-26 07:39:19.745517+00 > > > > lsub28_slot | 2024-03-26 07:40:24.953826+00 > > > > > > > > standby: > > > > > > > > postgres=# select slot_name,inactive_since from pg_replication_slots where failover = 't'; > > > > slot_name | inactive_since > > > > -------------+------------------------------- > > > > lsub27_slot | 2024-03-26 07:43:56.387324+00 > > > > lsub28_slot | 2024-03-26 07:43:56.387338+00 > > > > > > > > I don't think that should be the case. > > > > > > > > > > But why? This is exactly what we discussed in another thread where we > > > agreed to update inactive_since even for sync slots. > > > > Hum, I thought we agreed to "sync" it and to "update it to current time" > > only at promotion time. > > I think there may have been some misunderstanding here. Indeed ;-) > But now if I > rethink this, I am fine with 'inactive_since' getting synced from > primary to standby. But if we do that, we need to add docs stating > "inactive_since" represents primary's inactivity and not standby's > slots inactivity for synced slots. Yeah sure. > The reason for this clarification > is that the synced slot might be generated much later, yet > 'inactive_since' is synced from the primary, potentially indicating a > time considerably earlier than when the synced slot was actually > created. Right. > Another approach could be that "inactive_since" for synced slot > actually gives its own inactivity data rather than giving primary's > slot data. We update inactive_since on standby only at 3 occasions: > 1) at the time of creation of the synced slot. > 2) during standby restart. > 3) during promotion of standby. > > I have attached a sample patch for this idea as.txt file. Thanks! > I am fine with any of these approaches. One gives data synced from > primary for synced slots, while another gives actual inactivity data > of synced slots. What about another approach?: inactive_since gives data synced from primary for synced slots and another dedicated field (could be added later...) could represent what you suggest as the other option. Another cons of updating inactive_since at the current time during each slot sync cycle is that calling GetCurrentTimestamp() very frequently (during each sync cycle of very active slots) could be too costly. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 10:47 shveta malik <[email protected]> parent: Bertrand Drouvot <[email protected]> 0 siblings, 2 replies; 46+ messages in thread From: shveta malik @ 2024-03-26 10:47 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bharath Rupireddy <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Tue, Mar 26, 2024 at 3:50 PM Bertrand Drouvot <[email protected]> wrote: > > Hi, > > > I think there may have been some misunderstanding here. > > Indeed ;-) > > > But now if I > > rethink this, I am fine with 'inactive_since' getting synced from > > primary to standby. But if we do that, we need to add docs stating > > "inactive_since" represents primary's inactivity and not standby's > > slots inactivity for synced slots. > > Yeah sure. > > > The reason for this clarification > > is that the synced slot might be generated much later, yet > > 'inactive_since' is synced from the primary, potentially indicating a > > time considerably earlier than when the synced slot was actually > > created. > > Right. > > > Another approach could be that "inactive_since" for synced slot > > actually gives its own inactivity data rather than giving primary's > > slot data. We update inactive_since on standby only at 3 occasions: > > 1) at the time of creation of the synced slot. > > 2) during standby restart. > > 3) during promotion of standby. > > > > I have attached a sample patch for this idea as.txt file. > > Thanks! > > > I am fine with any of these approaches. One gives data synced from > > primary for synced slots, while another gives actual inactivity data > > of synced slots. > > What about another approach?: inactive_since gives data synced from primary for > synced slots and another dedicated field (could be added later...) could > represent what you suggest as the other option. Yes, okay with me. I think there is some confusion here as well. In my second approach above, I have not suggested anything related to sync-worker. We can think on that later if we really need another field which give us sync time. In my second approach, I have tried to avoid updating inactive_since for synced slots during sync process. We update that field during creation of synced slot so that inactive_since reflects correct info even for synced slots (rather than copying from primary). Please have a look at my patch and let me know your thoughts. I am fine with copying it from primary as well and documenting this behaviour. > Another cons of updating inactive_since at the current time during each slot > sync cycle is that calling GetCurrentTimestamp() very frequently > (during each sync cycle of very active slots) could be too costly. Right. thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 11:05 Bharath Rupireddy <[email protected]> parent: shveta malik <[email protected]> 1 sibling, 2 replies; 46+ messages in thread From: Bharath Rupireddy @ 2024-03-26 11:05 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Mar 26, 2024 at 4:18 PM shveta malik <[email protected]> wrote: > > > What about another approach?: inactive_since gives data synced from primary for > > synced slots and another dedicated field (could be added later...) could > > represent what you suggest as the other option. > > Yes, okay with me. I think there is some confusion here as well. In my > second approach above, I have not suggested anything related to > sync-worker. We can think on that later if we really need another > field which give us sync time. In my second approach, I have tried to > avoid updating inactive_since for synced slots during sync process. We > update that field during creation of synced slot so that > inactive_since reflects correct info even for synced slots (rather > than copying from primary). Please have a look at my patch and let me > know your thoughts. I am fine with copying it from primary as well and > documenting this behaviour. I took a look at your patch. --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -628,6 +628,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) SpinLockAcquire(&slot->mutex); slot->effective_catalog_xmin = xmin_horizon; slot->data.catalog_xmin = xmin_horizon; + slot->inactive_since = GetCurrentTimestamp(); SpinLockRelease(&slot->mutex); If we just sync inactive_since value for synced slots while in recovery from the primary, so be it. Why do we need to update it to the current time when the slot is being created? We don't expose slot creation time, no? Aren't we fine if we just sync the value from primary and document that fact? After the promotion, we can reset it to the current time so that it gets its own time. Do you see any issues with it? -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 11:19 shveta malik <[email protected]> parent: Bharath Rupireddy <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: shveta malik @ 2024-03-26 11:19 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Tue, Mar 26, 2024 at 4:35 PM Bharath Rupireddy <[email protected]> wrote: > > On Tue, Mar 26, 2024 at 4:18 PM shveta malik <[email protected]> wrote: > > > > > What about another approach?: inactive_since gives data synced from primary for > > > synced slots and another dedicated field (could be added later...) could > > > represent what you suggest as the other option. > > > > Yes, okay with me. I think there is some confusion here as well. In my > > second approach above, I have not suggested anything related to > > sync-worker. We can think on that later if we really need another > > field which give us sync time. In my second approach, I have tried to > > avoid updating inactive_since for synced slots during sync process. We > > update that field during creation of synced slot so that > > inactive_since reflects correct info even for synced slots (rather > > than copying from primary). Please have a look at my patch and let me > > know your thoughts. I am fine with copying it from primary as well and > > documenting this behaviour. > > I took a look at your patch. > > --- a/src/backend/replication/logical/slotsync.c > +++ b/src/backend/replication/logical/slotsync.c > @@ -628,6 +628,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid > remote_dbid) > SpinLockAcquire(&slot->mutex); > slot->effective_catalog_xmin = xmin_horizon; > slot->data.catalog_xmin = xmin_horizon; > + slot->inactive_since = GetCurrentTimestamp(); > SpinLockRelease(&slot->mutex); > > If we just sync inactive_since value for synced slots while in > recovery from the primary, so be it. Why do we need to update it to > the current time when the slot is being created? If we update inactive_since at synced slot's creation or during restart (skipping setting it during sync), then this time reflects actual 'inactive_since' for that particular synced slot. Isn't that a clear info for the user and in alignment of what the name 'inactive_since' actually suggests? > We don't expose slot > creation time, no? No, we don't. But for synced slot, that is the time since that slot is inactive (unless promoted), so we are exposing inactive_since and not creation time. >Aren't we fine if we just sync the value from > primary and document that fact? After the promotion, we can reset it > to the current time so that it gets its own time. Do you see any > issues with it? Yes, we can do that. But curious to know, do we see any additional benefit of reflecting primary's inactive_since at standby which I might be missing? thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 12:31 Bertrand Drouvot <[email protected]> parent: shveta malik <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-26 12:31 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Tue, Mar 26, 2024 at 04:49:18PM +0530, shveta malik wrote: > On Tue, Mar 26, 2024 at 4:35 PM Bharath Rupireddy > <[email protected]> wrote: > > > > On Tue, Mar 26, 2024 at 4:18 PM shveta malik <[email protected]> wrote: > > > > > > > What about another approach?: inactive_since gives data synced from primary for > > > > synced slots and another dedicated field (could be added later...) could > > > > represent what you suggest as the other option. > > > > > > Yes, okay with me. I think there is some confusion here as well. In my > > > second approach above, I have not suggested anything related to > > > sync-worker. We can think on that later if we really need another > > > field which give us sync time. In my second approach, I have tried to > > > avoid updating inactive_since for synced slots during sync process. We > > > update that field during creation of synced slot so that > > > inactive_since reflects correct info even for synced slots (rather > > > than copying from primary). Please have a look at my patch and let me > > > know your thoughts. I am fine with copying it from primary as well and > > > documenting this behaviour. > > > > I took a look at your patch. > > > > --- a/src/backend/replication/logical/slotsync.c > > +++ b/src/backend/replication/logical/slotsync.c > > @@ -628,6 +628,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid > > remote_dbid) > > SpinLockAcquire(&slot->mutex); > > slot->effective_catalog_xmin = xmin_horizon; > > slot->data.catalog_xmin = xmin_horizon; > > + slot->inactive_since = GetCurrentTimestamp(); > > SpinLockRelease(&slot->mutex); > > > > If we just sync inactive_since value for synced slots while in > > recovery from the primary, so be it. Why do we need to update it to > > the current time when the slot is being created? > > If we update inactive_since at synced slot's creation or during > restart (skipping setting it during sync), then this time reflects > actual 'inactive_since' for that particular synced slot. Isn't that a > clear info for the user and in alignment of what the name > 'inactive_since' actually suggests? > > > We don't expose slot > > creation time, no? > > No, we don't. But for synced slot, that is the time since that slot is > inactive (unless promoted), so we are exposing inactive_since and not > creation time. > > >Aren't we fine if we just sync the value from > > primary and document that fact? After the promotion, we can reset it > > to the current time so that it gets its own time. Do you see any > > issues with it? > > Yes, we can do that. But curious to know, do we see any additional > benefit of reflecting primary's inactive_since at standby which I > might be missing? In case the primary goes down, then one could use the value on the standby to get the value coming from the primary. I think that could be useful info to have. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 12:35 Bertrand Drouvot <[email protected]> parent: shveta malik <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-26 12:35 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bharath Rupireddy <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Tue, Mar 26, 2024 at 04:17:53PM +0530, shveta malik wrote: > On Tue, Mar 26, 2024 at 3:50 PM Bertrand Drouvot > <[email protected]> wrote: > > > > Hi, > > > > > I think there may have been some misunderstanding here. > > > > Indeed ;-) > > > > > But now if I > > > rethink this, I am fine with 'inactive_since' getting synced from > > > primary to standby. But if we do that, we need to add docs stating > > > "inactive_since" represents primary's inactivity and not standby's > > > slots inactivity for synced slots. > > > > Yeah sure. > > > > > The reason for this clarification > > > is that the synced slot might be generated much later, yet > > > 'inactive_since' is synced from the primary, potentially indicating a > > > time considerably earlier than when the synced slot was actually > > > created. > > > > Right. > > > > > Another approach could be that "inactive_since" for synced slot > > > actually gives its own inactivity data rather than giving primary's > > > slot data. We update inactive_since on standby only at 3 occasions: > > > 1) at the time of creation of the synced slot. > > > 2) during standby restart. > > > 3) during promotion of standby. > > > > > > I have attached a sample patch for this idea as.txt file. > > > > Thanks! > > > > > I am fine with any of these approaches. One gives data synced from > > > primary for synced slots, while another gives actual inactivity data > > > of synced slots. > > > > What about another approach?: inactive_since gives data synced from primary for > > synced slots and another dedicated field (could be added later...) could > > represent what you suggest as the other option. > > Yes, okay with me. I think there is some confusion here as well. In my > second approach above, I have not suggested anything related to > sync-worker. Yeah, no confusion, understood that way. > We can think on that later if we really need another > field which give us sync time. I think that calling GetCurrentTimestamp() so frequently could be too costly, so I'm not sure we should. > In my second approach, I have tried to > avoid updating inactive_since for synced slots during sync process. We > update that field during creation of synced slot so that > inactive_since reflects correct info even for synced slots (rather > than copying from primary). Yeah, and I think we could create a dedicated field with this information if we feel the need. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 16:29 Bharath Rupireddy <[email protected]> parent: Bharath Rupireddy <[email protected]> 1 sibling, 2 replies; 46+ messages in thread From: Bharath Rupireddy @ 2024-03-26 16:29 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Mar 26, 2024 at 4:35 PM Bharath Rupireddy <[email protected]> wrote: > > If we just sync inactive_since value for synced slots while in > recovery from the primary, so be it. Why do we need to update it to > the current time when the slot is being created? We don't expose slot > creation time, no? Aren't we fine if we just sync the value from > primary and document that fact? After the promotion, we can reset it > to the current time so that it gets its own time. I'm attaching v24 patches. It implements the above idea proposed upthread for synced slots. I've now separated s/last_inactive_time/inactive_since and synced slots behaviour. Please have a look. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com Attachments: [application/octet-stream] v24-0001-Use-less-confusing-name-for-slot-s-last_inactive.patch (14.3K, ../../CALj2ACUXS0SfbHzsX8bqo+7CZhocsV52Kiu7OWGb5HVPAmJqnA@mail.gmail.com/2-v24-0001-Use-less-confusing-name-for-slot-s-last_inactive.patch) download | inline diff: From 942b8742f108fadf776e3ae1f6fe800878d27b55 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Tue, 26 Mar 2024 16:17:55 +0000 Subject: [PATCH v24 1/4] Use less confusing name for slot's last_inactive_time property. The slot's last_inactive_time property added by commit a11f330b55 seems confusing. With last_inactive_time one expects it to tell the last time that the slot was inactive. But, it tells the last time that a currently-inactive slot previously *WAS* active. This commit uses a less confusing and better name for the property called inactive_since. Othernames considered were released_time, deactivated_at but inactive_since won the race since the word inactive is predominant as far as the replication slots are concerned. Reported-by: Robert Haas Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila Discussion: https://www.postgresql.org/message-id/ZgGrCBQoktdLi1Ir%40ip-10-97-1-34.eu-west-3.compute.internal --- doc/src/sgml/system-views.sgml | 4 +- src/backend/catalog/system_views.sql | 2 +- src/backend/replication/slot.c | 17 ++++--- src/backend/replication/slotfuncs.c | 4 +- src/include/catalog/pg_proc.dat | 2 +- src/include/replication/slot.h | 4 +- src/test/recovery/t/019_replslot_limit.pl | 62 +++++++++++------------ src/test/regress/expected/rules.out | 4 +- 8 files changed, 51 insertions(+), 48 deletions(-) diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 5f4165a945..3c8dca8ca3 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2525,10 +2525,10 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <row> <entry role="catalog_table_entry"><para role="column_definition"> - <structfield>last_inactive_time</structfield> <type>timestamptz</type> + <structfield>inactive_since</structfield> <type>timestamptz</type> </para> <para> - The time at which the slot became inactive. + The time since the slot has become inactive. <literal>NULL</literal> if the slot is currently being used. </para></entry> </row> diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index bc70ff193e..401fb35947 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1023,7 +1023,7 @@ CREATE VIEW pg_replication_slots AS L.wal_status, L.safe_wal_size, L.two_phase, - L.last_inactive_time, + L.inactive_since, L.conflicting, L.invalidation_reason, L.failover, diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 45f7a28f7d..d778c0b921 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -409,7 +409,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, slot->candidate_restart_valid = InvalidXLogRecPtr; slot->candidate_restart_lsn = InvalidXLogRecPtr; slot->last_saved_confirmed_flush = InvalidXLogRecPtr; - slot->last_inactive_time = 0; + slot->inactive_since = 0; /* * Create the slot on disk. We haven't actually marked the slot allocated @@ -623,9 +623,12 @@ retry: if (SlotIsLogical(s)) pgstat_acquire_replslot(s); - /* Reset the last inactive time as the slot is active now. */ + /* + * Reset the time since the slot has become inactive as the slot is active + * now. + */ SpinLockAcquire(&s->mutex); - s->last_inactive_time = 0; + s->inactive_since = 0; SpinLockRelease(&s->mutex); if (am_walsender) @@ -703,14 +706,14 @@ ReplicationSlotRelease(void) */ SpinLockAcquire(&slot->mutex); slot->active_pid = 0; - slot->last_inactive_time = now; + slot->inactive_since = now; SpinLockRelease(&slot->mutex); ConditionVariableBroadcast(&slot->active_cv); } else { SpinLockAcquire(&slot->mutex); - slot->last_inactive_time = now; + slot->inactive_since = now; SpinLockRelease(&slot->mutex); } @@ -2373,9 +2376,9 @@ RestoreSlotFromDisk(const char *name) * inactive as decoding is not allowed on those. */ if (!(RecoveryInProgress() && slot->data.synced)) - slot->last_inactive_time = GetCurrentTimestamp(); + slot->inactive_since = GetCurrentTimestamp(); else - slot->last_inactive_time = 0; + slot->inactive_since = 0; restored = true; break; diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index 24f5e6d90a..da57177c25 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -410,8 +410,8 @@ pg_get_replication_slots(PG_FUNCTION_ARGS) values[i++] = BoolGetDatum(slot_contents.data.two_phase); - if (slot_contents.last_inactive_time > 0) - values[i++] = TimestampTzGetDatum(slot_contents.last_inactive_time); + if (slot_contents.inactive_since > 0) + values[i++] = TimestampTzGetDatum(slot_contents.inactive_since); else nulls[i++] = true; diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 0d26e5b422..2f7cfc02c6 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11135,7 +11135,7 @@ proargtypes => '', proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,timestamptz,bool,text,bool,bool}', proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,last_inactive_time,conflicting,invalidation_reason,failover,synced}', + proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,inactive_since,conflicting,invalidation_reason,failover,synced}', prosrc => 'pg_get_replication_slots' }, { oid => '3786', descr => 'set up a logical replication slot', proname => 'pg_create_logical_replication_slot', provolatile => 'v', diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index eefd7abd39..7b937d1a0c 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -202,8 +202,8 @@ typedef struct ReplicationSlot */ XLogRecPtr last_saved_confirmed_flush; - /* The time at which this slot becomes inactive */ - TimestampTz last_inactive_time; + /* The time since the slot has become inactive */ + TimestampTz inactive_since; } ReplicationSlot; #define SlotIsPhysical(slot) ((slot)->data.database == InvalidOid) diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl index 3409cf88cd..3b9a306a8b 100644 --- a/src/test/recovery/t/019_replslot_limit.pl +++ b/src/test/recovery/t/019_replslot_limit.pl @@ -411,7 +411,7 @@ $node_primary3->stop; $node_standby3->stop; # ============================================================================= -# Testcase start: Check last_inactive_time property of the streaming standby's slot +# Testcase start: Check inactive_since property of the streaming standby's slot # # Initialize primary node @@ -440,45 +440,45 @@ $primary4->safe_psql( SELECT pg_create_physical_replication_slot(slot_name := '$sb4_slot'); ]); -# Get last_inactive_time value after the slot's creation. Note that the slot -# is still inactive till it's used by the standby below. -my $last_inactive_time = - capture_and_validate_slot_last_inactive_time($primary4, $sb4_slot, $slot_creation_time); +# Get inactive_since value after the slot's creation. Note that the slot is +# still inactive till it's used by the standby below. +my $inactive_since = + capture_and_validate_slot_inactive_since($primary4, $sb4_slot, $slot_creation_time); $standby4->start; # Wait until standby has replayed enough data $primary4->wait_for_catchup($standby4); -# Now the slot is active so last_inactive_time value must be NULL +# Now the slot is active so inactive_since value must be NULL is( $primary4->safe_psql( 'postgres', - qq[SELECT last_inactive_time IS NULL FROM pg_replication_slots WHERE slot_name = '$sb4_slot';] + qq[SELECT inactive_since IS NULL FROM pg_replication_slots WHERE slot_name = '$sb4_slot';] ), 't', 'last inactive time for an active physical slot is NULL'); -# Stop the standby to check its last_inactive_time value is updated +# Stop the standby to check its inactive_since value is updated $standby4->stop; -# Let's restart the primary so that the last_inactive_time is set upon -# loading the slot from the disk. +# Let's restart the primary so that the inactive_since is set upon loading the +# slot from the disk. $primary4->restart; is( $primary4->safe_psql( 'postgres', - qq[SELECT last_inactive_time > '$last_inactive_time'::timestamptz FROM pg_replication_slots WHERE slot_name = '$sb4_slot' AND last_inactive_time IS NOT NULL;] + qq[SELECT inactive_since > '$inactive_since'::timestamptz FROM pg_replication_slots WHERE slot_name = '$sb4_slot' AND inactive_since IS NOT NULL;] ), 't', 'last inactive time for an inactive physical slot is updated correctly'); $standby4->stop; -# Testcase end: Check last_inactive_time property of the streaming standby's slot +# Testcase end: Check inactive_since property of the streaming standby's slot # ============================================================================= # ============================================================================= -# Testcase start: Check last_inactive_time property of the logical subscriber's slot +# Testcase start: Check inactive_since property of the logical subscriber's slot my $publisher4 = $primary4; # Create subscriber node @@ -499,10 +499,10 @@ $publisher4->safe_psql('postgres', "SELECT pg_create_logical_replication_slot(slot_name := '$lsub4_slot', plugin := 'pgoutput');" ); -# Get last_inactive_time value after the slot's creation. Note that the slot -# is still inactive till it's used by the subscriber below. -$last_inactive_time = - capture_and_validate_slot_last_inactive_time($publisher4, $lsub4_slot, $slot_creation_time); +# Get inactive_since value after the slot's creation. Note that the slot is +# still inactive till it's used by the subscriber below. +$inactive_since = + capture_and_validate_slot_inactive_since($publisher4, $lsub4_slot, $slot_creation_time); $subscriber4->start; $subscriber4->safe_psql('postgres', @@ -512,54 +512,54 @@ $subscriber4->safe_psql('postgres', # Wait until subscriber has caught up $subscriber4->wait_for_subscription_sync($publisher4, 'sub'); -# Now the slot is active so last_inactive_time value must be NULL +# Now the slot is active so inactive_since value must be NULL is( $publisher4->safe_psql( 'postgres', - qq[SELECT last_inactive_time IS NULL FROM pg_replication_slots WHERE slot_name = '$lsub4_slot';] + qq[SELECT inactive_since IS NULL FROM pg_replication_slots WHERE slot_name = '$lsub4_slot';] ), 't', 'last inactive time for an active logical slot is NULL'); -# Stop the subscriber to check its last_inactive_time value is updated +# Stop the subscriber to check its inactive_since value is updated $subscriber4->stop; -# Let's restart the publisher so that the last_inactive_time is set upon +# Let's restart the publisher so that the inactive_since is set upon # loading the slot from the disk. $publisher4->restart; is( $publisher4->safe_psql( 'postgres', - qq[SELECT last_inactive_time > '$last_inactive_time'::timestamptz FROM pg_replication_slots WHERE slot_name = '$lsub4_slot' AND last_inactive_time IS NOT NULL;] + qq[SELECT inactive_since > '$inactive_since'::timestamptz FROM pg_replication_slots WHERE slot_name = '$lsub4_slot' AND inactive_since IS NOT NULL;] ), 't', 'last inactive time for an inactive logical slot is updated correctly'); -# Testcase end: Check last_inactive_time property of the logical subscriber's slot +# Testcase end: Check inactive_since property of the logical subscriber's slot # ============================================================================= $publisher4->stop; $subscriber4->stop; -# Capture and validate last_inactive_time of a given slot. -sub capture_and_validate_slot_last_inactive_time +# Capture and validate inactive_since of a given slot. +sub capture_and_validate_slot_inactive_since { my ($node, $slot_name, $slot_creation_time) = @_; - my $last_inactive_time = $node->safe_psql('postgres', - qq(SELECT last_inactive_time FROM pg_replication_slots - WHERE slot_name = '$slot_name' AND last_inactive_time IS NOT NULL;) + my $inactive_since = $node->safe_psql('postgres', + qq(SELECT inactive_since FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) ); # Check that the captured time is sane is( $node->safe_psql( 'postgres', - qq[SELECT '$last_inactive_time'::timestamptz > to_timestamp(0) AND - '$last_inactive_time'::timestamptz >= '$slot_creation_time'::timestamptz;] + qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND + '$inactive_since'::timestamptz >= '$slot_creation_time'::timestamptz;] ), 't', "last inactive time for an active slot $slot_name is sane"); - return $last_inactive_time; + return $inactive_since; } done_testing(); diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index dfcbaec387..f53c3036a6 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1473,12 +1473,12 @@ pg_replication_slots| SELECT l.slot_name, l.wal_status, l.safe_wal_size, l.two_phase, - l.last_inactive_time, + l.inactive_since, l.conflicting, l.invalidation_reason, l.failover, l.synced - FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, last_inactive_time, conflicting, invalidation_reason, failover, synced) + FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, inactive_since, conflicting, invalidation_reason, failover, synced) LEFT JOIN pg_database d ON ((l.datoid = d.oid))); pg_roles| SELECT pg_authid.rolname, pg_authid.rolsuper, -- 2.34.1 [application/octet-stream] v24-0002-Maintain-inactive_since-for-synced-slots-correct.patch (11.9K, ../../CALj2ACUXS0SfbHzsX8bqo+7CZhocsV52Kiu7OWGb5HVPAmJqnA@mail.gmail.com/3-v24-0002-Maintain-inactive_since-for-synced-slots-correct.patch) download | inline diff: From d9df4b9f3fa146f076064cdf10cfefc6a97ccca2 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Tue, 26 Mar 2024 16:18:58 +0000 Subject: [PATCH v24 2/4] Maintain inactive_since for synced slots correctly. The slot's inactive_since isn't currently maintained for synced slots on the standby. The commit a11f330b55 prevents updating inactive_since with RecoveryInProgress() check in RestoreSlotFromDisk(). But, the issue is that RecoveryInProgress() always returns true in RestoreSlotFromDisk() as 'xlogctl->SharedRecoveryState' is always 'RECOVERY_STATE_CRASH' at that time. The impact of this on a promoted standby inactive_since is always NULL for all synced slots even after server restart. Above issue led us to a question as to why we can't just update inactive_since for synced slots on the standby with the value received from remote slot on the primary. This is consistent with any other slot parameter i.e. all of them are synced from the primary. This commit does two things: 1) Updates inactive_since for sync slots with the value received from the primary's slot. 2) Ensures the value is set to current timestamp during the shutdown of slot sync machinery to help correctly interpret the time if the standby gets promoted without a restart. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACWLctoiH-pSjWnEpR54q4DED6rw_BRJm5pCx86_Y01MoQ%40mail.gmail.com --- src/backend/replication/logical/slotsync.c | 60 +++++++++++++++++-- src/backend/replication/slot.c | 37 ++++++++---- .../t/040_standby_failover_slots_sync.pl | 49 +++++++++++++++ 3 files changed, 130 insertions(+), 16 deletions(-) diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 30480960c5..c1905ce24b 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -137,9 +137,12 @@ typedef struct RemoteSlot /* RS_INVAL_NONE if valid, or the reason of invalidation */ ReplicationSlotInvalidationCause invalidated; + + TimestampTz inactive_since; /* in seconds */ } RemoteSlot; static void slotsync_failure_callback(int code, Datum arg); +static void update_synced_slots_inactive_time(void); /* * If necessary, update the local synced slot's metadata based on the data @@ -167,7 +170,8 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) remote_slot->two_phase == slot->data.two_phase && remote_slot->failover == slot->data.failover && remote_slot->confirmed_lsn == slot->data.confirmed_flush && - strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) + strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0 && + remote_slot->inactive_since == slot->inactive_since) return false; /* Avoid expensive operations while holding a spinlock. */ @@ -182,6 +186,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) slot->data.confirmed_flush = remote_slot->confirmed_lsn; slot->data.catalog_xmin = remote_slot->catalog_xmin; slot->effective_catalog_xmin = remote_slot->catalog_xmin; + slot->inactive_since = remote_slot->inactive_since; SpinLockRelease(&slot->mutex); if (xmin_changed) @@ -652,9 +657,9 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) static bool synchronize_slots(WalReceiverConn *wrconn) { -#define SLOTSYNC_COLUMN_COUNT 9 +#define SLOTSYNC_COLUMN_COUNT 10 Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, - LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID, TIMESTAMPTZOID}; WalRcvExecResult *res; TupleTableSlot *tupslot; @@ -663,7 +668,7 @@ synchronize_slots(WalReceiverConn *wrconn) bool started_tx = false; const char *query = "SELECT slot_name, plugin, confirmed_flush_lsn," " restart_lsn, catalog_xmin, two_phase, failover," - " database, invalidation_reason" + " database, invalidation_reason, inactive_since" " FROM pg_catalog.pg_replication_slots" " WHERE failover and NOT temporary"; @@ -743,6 +748,9 @@ synchronize_slots(WalReceiverConn *wrconn) remote_slot->invalidated = isnull ? RS_INVAL_NONE : GetSlotInvalidationCause(TextDatumGetCString(d)); + remote_slot->inactive_since = DatumGetTimestampTz(slot_getattr(tupslot, ++col, + &isnull)); + /* Sanity check */ Assert(col == SLOTSYNC_COLUMN_COUNT); @@ -1296,6 +1304,47 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len) Assert(false); } +/* + * Reset the synced slots info such as inactive_since after shutting + * down the slot sync machinery. + */ +static void +update_synced_slots_inactive_time(void) +{ + TimestampTz now = 0; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + + /* + * We get the current time beforehand and only once to avoid + * system calls overhead while holding the lock. + */ + if (now == 0) + now = GetCurrentTimestamp(); + + /* + * Set the time since the slot has become inactive after shutting + * down slot sync machinery. This helps correctly interpret the + * time if the standby gets promoted without a restart. + */ + SpinLockAcquire(&s->mutex); + s->inactive_since = now; + SpinLockRelease(&s->mutex); + } + } + + LWLockRelease(ReplicationSlotControlLock); +} + /* * Shut down the slot sync worker. */ @@ -1309,6 +1358,7 @@ ShutDownSlotSync(void) if (SlotSyncCtx->pid == InvalidPid) { SpinLockRelease(&SlotSyncCtx->mutex); + update_synced_slots_inactive_time(); return; } SpinLockRelease(&SlotSyncCtx->mutex); @@ -1341,6 +1391,8 @@ ShutDownSlotSync(void) } SpinLockRelease(&SlotSyncCtx->mutex); + + update_synced_slots_inactive_time(); } /* diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index d778c0b921..05bc453de9 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -42,6 +42,7 @@ #include "access/transam.h" #include "access/xlog_internal.h" #include "access/xlogrecovery.h" +#include "access/xlogutils.h" #include "common/file_utils.h" #include "common/string.h" #include "miscadmin.h" @@ -655,6 +656,7 @@ ReplicationSlotRelease(void) char *slotname = NULL; /* keep compiler quiet */ bool is_logical = false; /* keep compiler quiet */ TimestampTz now = 0; + bool update_inactive_since; Assert(slot != NULL && slot->active_pid != 0); @@ -690,13 +692,19 @@ ReplicationSlotRelease(void) } /* - * Set the last inactive time after marking the slot inactive. We don't - * set it for the slots currently being synced from the primary to the - * standby because such slots are typically inactive as decoding is not - * allowed on those. + * Set the time since the slot has become inactive. + * + * Note that we don't set it for the slots currently being synced from the + * primary to the standby, because such slots typically sync the data from + * the remote slot. */ if (!(RecoveryInProgress() && slot->data.synced)) + { now = GetCurrentTimestamp(); + update_inactive_since = true; + } + else + update_inactive_since = false; if (slot->data.persistency == RS_PERSISTENT) { @@ -706,11 +714,12 @@ ReplicationSlotRelease(void) */ SpinLockAcquire(&slot->mutex); slot->active_pid = 0; - slot->inactive_since = now; + if (update_inactive_since) + slot->inactive_since = now; SpinLockRelease(&slot->mutex); ConditionVariableBroadcast(&slot->active_cv); } - else + else if (update_inactive_since) { SpinLockAcquire(&slot->mutex); slot->inactive_since = now; @@ -2369,13 +2378,17 @@ RestoreSlotFromDisk(const char *name) slot->active_pid = 0; /* - * We set the last inactive time after loading the slot from the disk - * into memory. Whoever acquires the slot i.e. makes the slot active - * will reset it. We don't set it for the slots currently being synced - * from the primary to the standby because such slots are typically - * inactive as decoding is not allowed on those. + * Set the time since the slot has become inactive after loading the + * slot from the disk into memory. Whoever acquires the slot i.e. + * makes the slot active will reset it. + * + * Note that we don't set it for the slots currently being synced from + * the primary to the standby, because such slots typically sync the + * data from the remote slot. We use InRecovery flag instead of + * RecoveryInProgress() as it always returns true even for normal + * server startup. */ - if (!(RecoveryInProgress() && slot->data.synced)) + if (!(InRecovery && slot->data.synced)) slot->inactive_since = GetCurrentTimestamp(); else slot->inactive_since = 0; diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index f47bfd78eb..a5aa2e3260 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1'); $subscriber1->init; $subscriber1->start; +# Capture the time before the logical failover slot is created on the +# primary. We later call this publisher as primary anyway. +my $creation_time_on_primary = $publisher->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + # Create a slot on the publisher with failover disabled $publisher->safe_psql('postgres', "SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);" @@ -174,6 +181,10 @@ $primary->poll_query_until( "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", 1); +# Capture the inactive_since of the slot from the primary +my $inactive_since_on_primary = + capture_and_validate_slot_inactive_since($primary, 'lsub1_slot', $creation_time_on_primary); + # Wait for the standby to catch up so that the standby is not lagging behind # the subscriber. $primary->wait_for_replay_catchup($standby1); @@ -190,6 +201,19 @@ is( $standby1->safe_psql( "t", 'logical slots have synced as true on standby'); +# Capture the inactive_since of the slot from the standby the logical failover +# slots are synced/created on the standby. +my $inactive_since_on_standby = + capture_and_validate_slot_inactive_since($standby1, 'lsub1_slot', $creation_time_on_primary); + +# Synced slots on the standby must get the inactive_since from the primary. +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_primary'::timestamptz = '$inactive_since_on_standby'::timestamptz;" + ), + "t", + 'synchronized slot has got the inactive_since from the primary'); + ################################################## # Test that the synchronized slot will be dropped if the corresponding remote # slot on the primary server has been dropped. @@ -773,4 +797,29 @@ is( $subscriber1->safe_psql('postgres', q{SELECT count(*) FROM tab_int;}), "20", 'data replicated from the new primary'); +# Capture and validate inactive_since of a given slot. +sub capture_and_validate_slot_inactive_since +{ + my ($node, $slot_name, $slot_creation_time) = @_; + my $name = $node->name; + + my $inactive_since = $node->safe_psql('postgres', + qq(SELECT inactive_since FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) + ); + + print "HI $slot_name $name $inactive_since $slot_creation_time\n"; + + # Check that the captured time is sane + is( $node->safe_psql( + 'postgres', + qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND + '$inactive_since'::timestamptz >= '$slot_creation_time'::timestamptz;] + ), + 't', + "last inactive time for slot $slot_name is sane on node $name"); + + return $inactive_since; +} + done_testing(); -- 2.34.1 [application/octet-stream] v24-0003-Allow-setting-inactive_timeout-for-replication-s.patch (34.6K, ../../CALj2ACUXS0SfbHzsX8bqo+7CZhocsV52Kiu7OWGb5HVPAmJqnA@mail.gmail.com/4-v24-0003-Allow-setting-inactive_timeout-for-replication-s.patch) download | inline diff: From 8c6d41253ed2fd301cd062fb408660d26ec7c751 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Tue, 26 Mar 2024 16:19:35 +0000 Subject: [PATCH v24 3/4] Allow setting inactive_timeout for replication slots via SQL API. This commit adds a new replication slot property called inactive_timeout specifying the amount of time in seconds the slot is allowed to be inactive. It is added to slot's persistent data structure to survive during server restarts. It will be synced to failover slots on the standby, and also will be carried over to the new cluster as part of pg_upgrade. This commit particularly lets one specify the inactive_timeout for a slot via SQL functions pg_create_physical_replication_slot and pg_create_logical_replication_slot. The new property will be useful to implement inactive timeout based replication slot invalidation in a future commit. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com --- contrib/test_decoding/expected/slot.out | 97 +++++++++++++++++++ contrib/test_decoding/sql/slot.sql | 30 ++++++ doc/src/sgml/func.sgml | 18 ++-- doc/src/sgml/system-views.sgml | 9 ++ src/backend/catalog/system_functions.sql | 2 + src/backend/catalog/system_views.sql | 1 + src/backend/replication/logical/slotsync.c | 17 +++- src/backend/replication/slot.c | 8 +- src/backend/replication/slotfuncs.c | 31 +++++- src/backend/replication/walsender.c | 4 +- src/bin/pg_upgrade/info.c | 6 +- src/bin/pg_upgrade/pg_upgrade.c | 5 +- src/bin/pg_upgrade/pg_upgrade.h | 2 + src/bin/pg_upgrade/t/003_logical_slots.pl | 11 ++- src/include/catalog/pg_proc.dat | 22 ++--- src/include/replication/slot.h | 5 +- .../t/040_standby_failover_slots_sync.pl | 13 ++- src/test/regress/expected/rules.out | 3 +- 18 files changed, 243 insertions(+), 41 deletions(-) diff --git a/contrib/test_decoding/expected/slot.out b/contrib/test_decoding/expected/slot.out index 349ab2d380..c318eceefd 100644 --- a/contrib/test_decoding/expected/slot.out +++ b/contrib/test_decoding/expected/slot.out @@ -466,3 +466,100 @@ SELECT pg_drop_replication_slot('physical_slot'); (1 row) +-- Test negative value for inactive_timeout option for slots. +SELECT 'init' FROM pg_create_physical_replication_slot(slot_name := 'it_fail_slot', inactive_timeout := -300); -- error +ERROR: "inactive_timeout" must not be negative +SELECT 'init' FROM pg_create_logical_replication_slot(slot_name := 'it_fail_slot', plugin := 'test_decoding', inactive_timeout := -600); -- error +ERROR: "inactive_timeout" must not be negative +-- Test inactive_timeout option of physical slots. +SELECT 'init' FROM pg_create_physical_replication_slot(slot_name := 'it_phy_slot1', immediately_reserve := true, inactive_timeout := 300); + ?column? +---------- + init +(1 row) + +SELECT 'init' FROM pg_create_physical_replication_slot(slot_name := 'it_phy_slot2'); + ?column? +---------- + init +(1 row) + +-- Copy physical slot with inactive_timeout option set. +SELECT 'copy' FROM pg_copy_physical_replication_slot(src_slot_name := 'it_phy_slot1', dst_slot_name := 'it_phy_slot3'); + ?column? +---------- + copy +(1 row) + +SELECT slot_name, slot_type, inactive_timeout FROM pg_replication_slots ORDER BY 1; + slot_name | slot_type | inactive_timeout +--------------+-----------+------------------ + it_phy_slot1 | physical | 300 + it_phy_slot2 | physical | 0 + it_phy_slot3 | physical | 300 +(3 rows) + +SELECT pg_drop_replication_slot('it_phy_slot1'); + pg_drop_replication_slot +-------------------------- + +(1 row) + +SELECT pg_drop_replication_slot('it_phy_slot2'); + pg_drop_replication_slot +-------------------------- + +(1 row) + +SELECT pg_drop_replication_slot('it_phy_slot3'); + pg_drop_replication_slot +-------------------------- + +(1 row) + +-- Test inactive_timeout option of logical slots. +SELECT 'init' FROM pg_create_logical_replication_slot(slot_name := 'it_log_slot1', plugin := 'test_decoding', inactive_timeout := 600); + ?column? +---------- + init +(1 row) + +SELECT 'init' FROM pg_create_logical_replication_slot(slot_name := 'it_log_slot2', plugin := 'test_decoding'); + ?column? +---------- + init +(1 row) + +-- Copy logical slot with inactive_timeout option set. +SELECT 'copy' FROM pg_copy_logical_replication_slot(src_slot_name := 'it_log_slot1', dst_slot_name := 'it_log_slot3'); + ?column? +---------- + copy +(1 row) + +SELECT slot_name, slot_type, inactive_timeout FROM pg_replication_slots ORDER BY 1; + slot_name | slot_type | inactive_timeout +--------------+-----------+------------------ + it_log_slot1 | logical | 600 + it_log_slot2 | logical | 0 + it_log_slot3 | logical | 600 +(3 rows) + +SELECT pg_drop_replication_slot('it_log_slot1'); + pg_drop_replication_slot +-------------------------- + +(1 row) + +SELECT pg_drop_replication_slot('it_log_slot2'); + pg_drop_replication_slot +-------------------------- + +(1 row) + +SELECT pg_drop_replication_slot('it_log_slot3'); + pg_drop_replication_slot +-------------------------- + +(1 row) + diff --git a/contrib/test_decoding/sql/slot.sql b/contrib/test_decoding/sql/slot.sql index 580e3ae3be..e5c7b3d359 100644 --- a/contrib/test_decoding/sql/slot.sql +++ b/contrib/test_decoding/sql/slot.sql @@ -190,3 +190,33 @@ SELECT pg_drop_replication_slot('failover_true_slot'); SELECT pg_drop_replication_slot('failover_false_slot'); SELECT pg_drop_replication_slot('failover_default_slot'); SELECT pg_drop_replication_slot('physical_slot'); + +-- Test negative value for inactive_timeout option for slots. +SELECT 'init' FROM pg_create_physical_replication_slot(slot_name := 'it_fail_slot', inactive_timeout := -300); -- error +SELECT 'init' FROM pg_create_logical_replication_slot(slot_name := 'it_fail_slot', plugin := 'test_decoding', inactive_timeout := -600); -- error + +-- Test inactive_timeout option of physical slots. +SELECT 'init' FROM pg_create_physical_replication_slot(slot_name := 'it_phy_slot1', immediately_reserve := true, inactive_timeout := 300); +SELECT 'init' FROM pg_create_physical_replication_slot(slot_name := 'it_phy_slot2'); + +-- Copy physical slot with inactive_timeout option set. +SELECT 'copy' FROM pg_copy_physical_replication_slot(src_slot_name := 'it_phy_slot1', dst_slot_name := 'it_phy_slot3'); + +SELECT slot_name, slot_type, inactive_timeout FROM pg_replication_slots ORDER BY 1; + +SELECT pg_drop_replication_slot('it_phy_slot1'); +SELECT pg_drop_replication_slot('it_phy_slot2'); +SELECT pg_drop_replication_slot('it_phy_slot3'); + +-- Test inactive_timeout option of logical slots. +SELECT 'init' FROM pg_create_logical_replication_slot(slot_name := 'it_log_slot1', plugin := 'test_decoding', inactive_timeout := 600); +SELECT 'init' FROM pg_create_logical_replication_slot(slot_name := 'it_log_slot2', plugin := 'test_decoding'); + +-- Copy logical slot with inactive_timeout option set. +SELECT 'copy' FROM pg_copy_logical_replication_slot(src_slot_name := 'it_log_slot1', dst_slot_name := 'it_log_slot3'); + +SELECT slot_name, slot_type, inactive_timeout FROM pg_replication_slots ORDER BY 1; + +SELECT pg_drop_replication_slot('it_log_slot1'); +SELECT pg_drop_replication_slot('it_log_slot2'); +SELECT pg_drop_replication_slot('it_log_slot3'); diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 8ecc02f2b9..2cc26e927a 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28373,7 +28373,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <indexterm> <primary>pg_create_physical_replication_slot</primary> </indexterm> - <function>pg_create_physical_replication_slot</function> ( <parameter>slot_name</parameter> <type>name</type> <optional>, <parameter>immediately_reserve</parameter> <type>boolean</type>, <parameter>temporary</parameter> <type>boolean</type> </optional> ) + <function>pg_create_physical_replication_slot</function> ( <parameter>slot_name</parameter> <type>name</type> <optional>, <parameter>immediately_reserve</parameter> <type>boolean</type>, <parameter>temporary</parameter> <type>boolean</type>, <parameter>inactive_timeout</parameter> <type>integer</type> </optional>) <returnvalue>record</returnvalue> ( <parameter>slot_name</parameter> <type>name</type>, <parameter>lsn</parameter> <type>pg_lsn</type> ) @@ -28390,9 +28390,12 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset parameter, <parameter>temporary</parameter>, when set to true, specifies that the slot should not be permanently stored to disk and is only meant for use by the current session. Temporary slots are also - released upon any error. This function corresponds - to the replication protocol command <literal>CREATE_REPLICATION_SLOT - ... PHYSICAL</literal>. + released upon any error. The optional fourth + parameter, <parameter>inactive_timeout</parameter>, when set to a + non-zero value, specifies the amount of time in seconds the slot is + allowed to be inactive. This function corresponds to the replication + protocol command + <literal>CREATE_REPLICATION_SLOT ... PHYSICAL</literal>. </para></entry> </row> @@ -28417,7 +28420,7 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <indexterm> <primary>pg_create_logical_replication_slot</primary> </indexterm> - <function>pg_create_logical_replication_slot</function> ( <parameter>slot_name</parameter> <type>name</type>, <parameter>plugin</parameter> <type>name</type> <optional>, <parameter>temporary</parameter> <type>boolean</type>, <parameter>twophase</parameter> <type>boolean</type>, <parameter>failover</parameter> <type>boolean</type> </optional> ) + <function>pg_create_logical_replication_slot</function> ( <parameter>slot_name</parameter> <type>name</type>, <parameter>plugin</parameter> <type>name</type> <optional>, <parameter>temporary</parameter> <type>boolean</type>, <parameter>twophase</parameter> <type>boolean</type>, <parameter>failover</parameter> <type>boolean</type>, <parameter>inactive_timeout</parameter> <type>integer</type> </optional> ) <returnvalue>record</returnvalue> ( <parameter>slot_name</parameter> <type>name</type>, <parameter>lsn</parameter> <type>pg_lsn</type> ) @@ -28436,7 +28439,10 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <parameter>failover</parameter>, when set to true, specifies that this slot is enabled to be synced to the standbys so that logical replication can be resumed after - failover. A call to this function has the same effect as + failover. The optional sixth parameter, + <parameter>inactive_timeout</parameter>, when set to a + non-zero value, specifies the amount of time in seconds the slot is + allowed to be inactive. A call to this function has the same effect as the replication protocol command <literal>CREATE_REPLICATION_SLOT ... LOGICAL</literal>. </para></entry> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 3c8dca8ca3..a6cb13fd9d 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2523,6 +2523,15 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>inactive_timeout</structfield> <type>integer</type> + </para> + <para> + The amount of time in seconds the slot is allowed to be inactive. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>inactive_since</structfield> <type>timestamptz</type> diff --git a/src/backend/catalog/system_functions.sql b/src/backend/catalog/system_functions.sql index fe2bb50f46..af27616657 100644 --- a/src/backend/catalog/system_functions.sql +++ b/src/backend/catalog/system_functions.sql @@ -469,6 +469,7 @@ AS 'pg_logical_emit_message_bytea'; CREATE OR REPLACE FUNCTION pg_create_physical_replication_slot( IN slot_name name, IN immediately_reserve boolean DEFAULT false, IN temporary boolean DEFAULT false, + IN inactive_timeout int DEFAULT 0, OUT slot_name name, OUT lsn pg_lsn) RETURNS RECORD LANGUAGE INTERNAL @@ -480,6 +481,7 @@ CREATE OR REPLACE FUNCTION pg_create_logical_replication_slot( IN temporary boolean DEFAULT false, IN twophase boolean DEFAULT false, IN failover boolean DEFAULT false, + IN inactive_timeout int DEFAULT 0, OUT slot_name name, OUT lsn pg_lsn) RETURNS RECORD LANGUAGE INTERNAL diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 401fb35947..7d9d743dd5 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1023,6 +1023,7 @@ CREATE VIEW pg_replication_slots AS L.wal_status, L.safe_wal_size, L.two_phase, + L.inactive_timeout, L.inactive_since, L.conflicting, L.invalidation_reason, diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index c1905ce24b..91c1604d3c 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -131,6 +131,7 @@ typedef struct RemoteSlot char *database; bool two_phase; bool failover; + int inactive_timeout; /* in seconds */ XLogRecPtr restart_lsn; XLogRecPtr confirmed_lsn; TransactionId catalog_xmin; @@ -171,7 +172,8 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) remote_slot->failover == slot->data.failover && remote_slot->confirmed_lsn == slot->data.confirmed_flush && strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0 && - remote_slot->inactive_since == slot->inactive_since) + remote_slot->inactive_since == slot->inactive_since && + remote_slot->inactive_timeout == slot->data.inactive_timeout) return false; /* Avoid expensive operations while holding a spinlock. */ @@ -187,6 +189,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) slot->data.catalog_xmin = remote_slot->catalog_xmin; slot->effective_catalog_xmin = remote_slot->catalog_xmin; slot->inactive_since = remote_slot->inactive_since; + slot->data.inactive_timeout = remote_slot->inactive_timeout; SpinLockRelease(&slot->mutex); if (xmin_changed) @@ -612,7 +615,8 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) ReplicationSlotCreate(remote_slot->name, true, RS_TEMPORARY, remote_slot->two_phase, remote_slot->failover, - true); + true, + remote_slot->inactive_timeout); /* For shorter lines. */ slot = MyReplicationSlot; @@ -657,9 +661,9 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) static bool synchronize_slots(WalReceiverConn *wrconn) { -#define SLOTSYNC_COLUMN_COUNT 10 +#define SLOTSYNC_COLUMN_COUNT 11 Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, - LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID, TIMESTAMPTZOID}; + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID, TIMESTAMPTZOID, INT4OID}; WalRcvExecResult *res; TupleTableSlot *tupslot; @@ -668,7 +672,7 @@ synchronize_slots(WalReceiverConn *wrconn) bool started_tx = false; const char *query = "SELECT slot_name, plugin, confirmed_flush_lsn," " restart_lsn, catalog_xmin, two_phase, failover," - " database, invalidation_reason, inactive_since" + " database, invalidation_reason, inactive_since, inactive_timeout" " FROM pg_catalog.pg_replication_slots" " WHERE failover and NOT temporary"; @@ -751,6 +755,9 @@ synchronize_slots(WalReceiverConn *wrconn) remote_slot->inactive_since = DatumGetTimestampTz(slot_getattr(tupslot, ++col, &isnull)); + remote_slot->inactive_timeout = DatumGetInt32(slot_getattr(tupslot, ++col, + &isnull)); + /* Sanity check */ Assert(col == SLOTSYNC_COLUMN_COUNT); diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 05bc453de9..5653e680a8 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -130,7 +130,7 @@ StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1), sizeof(ReplicationSlotOnDisk) - ReplicationSlotOnDiskConstantSize #define SLOT_MAGIC 0x1051CA1 /* format identifier */ -#define SLOT_VERSION 5 /* version for new files */ +#define SLOT_VERSION 6 /* version for new files */ /* Control array for replication slot management */ ReplicationSlotCtlData *ReplicationSlotCtl = NULL; @@ -305,11 +305,14 @@ ReplicationSlotValidateName(const char *name, int elevel) * failover: If enabled, allows the slot to be synced to standbys so * that logical replication can be resumed after failover. * synced: True if the slot is synchronized from the primary server. + * inactive_timeout: The amount of time in seconds the slot is allowed to be + * inactive. */ void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, - bool two_phase, bool failover, bool synced) + bool two_phase, bool failover, bool synced, + int inactive_timeout) { ReplicationSlot *slot = NULL; int i; @@ -399,6 +402,7 @@ ReplicationSlotCreate(const char *name, bool db_specific, slot->data.two_phase_at = InvalidXLogRecPtr; slot->data.failover = failover; slot->data.synced = synced; + slot->data.inactive_timeout = inactive_timeout; /* and then data only present in shared memory */ slot->just_dirtied = false; diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index da57177c25..6e1d8d1f9a 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -38,14 +38,15 @@ */ static void create_physical_replication_slot(char *name, bool immediately_reserve, - bool temporary, XLogRecPtr restart_lsn) + bool temporary, int inactive_timeout, + XLogRecPtr restart_lsn) { Assert(!MyReplicationSlot); /* acquire replication slot, this will check for conflicting names */ ReplicationSlotCreate(name, false, temporary ? RS_TEMPORARY : RS_PERSISTENT, false, - false, false); + false, false, inactive_timeout); if (immediately_reserve) { @@ -71,6 +72,7 @@ pg_create_physical_replication_slot(PG_FUNCTION_ARGS) Name name = PG_GETARG_NAME(0); bool immediately_reserve = PG_GETARG_BOOL(1); bool temporary = PG_GETARG_BOOL(2); + int inactive_timeout = PG_GETARG_INT32(3); /* in seconds */ Datum values[2]; bool nulls[2]; TupleDesc tupdesc; @@ -84,9 +86,15 @@ pg_create_physical_replication_slot(PG_FUNCTION_ARGS) CheckSlotRequirements(); + if (inactive_timeout < 0) + ereport(ERROR, + (errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE), + errmsg("\"inactive_timeout\" must not be negative"))); + create_physical_replication_slot(NameStr(*name), immediately_reserve, temporary, + inactive_timeout, InvalidXLogRecPtr); values[0] = NameGetDatum(&MyReplicationSlot->data.name); @@ -120,7 +128,7 @@ pg_create_physical_replication_slot(PG_FUNCTION_ARGS) static void create_logical_replication_slot(char *name, char *plugin, bool temporary, bool two_phase, - bool failover, + bool failover, int inactive_timeout, XLogRecPtr restart_lsn, bool find_startpoint) { @@ -138,7 +146,7 @@ create_logical_replication_slot(char *name, char *plugin, */ ReplicationSlotCreate(name, true, temporary ? RS_TEMPORARY : RS_EPHEMERAL, two_phase, - failover, false); + failover, false, inactive_timeout); /* * Create logical decoding context to find start point or, if we don't @@ -177,6 +185,7 @@ pg_create_logical_replication_slot(PG_FUNCTION_ARGS) bool temporary = PG_GETARG_BOOL(2); bool two_phase = PG_GETARG_BOOL(3); bool failover = PG_GETARG_BOOL(4); + int inactive_timeout = PG_GETARG_INT32(5); /* in seconds */ Datum result; TupleDesc tupdesc; HeapTuple tuple; @@ -190,11 +199,17 @@ pg_create_logical_replication_slot(PG_FUNCTION_ARGS) CheckLogicalDecodingRequirements(); + if (inactive_timeout < 0) + ereport(ERROR, + (errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE), + errmsg("\"inactive_timeout\" must not be negative"))); + create_logical_replication_slot(NameStr(*name), NameStr(*plugin), temporary, two_phase, failover, + inactive_timeout, InvalidXLogRecPtr, true); @@ -239,7 +254,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS) Datum pg_get_replication_slots(PG_FUNCTION_ARGS) { -#define PG_GET_REPLICATION_SLOTS_COLS 19 +#define PG_GET_REPLICATION_SLOTS_COLS 20 ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; XLogRecPtr currlsn; int slotno; @@ -410,6 +425,8 @@ pg_get_replication_slots(PG_FUNCTION_ARGS) values[i++] = BoolGetDatum(slot_contents.data.two_phase); + values[i++] = Int32GetDatum(slot_contents.data.inactive_timeout); + if (slot_contents.inactive_since > 0) values[i++] = TimestampTzGetDatum(slot_contents.inactive_since); else @@ -720,6 +737,7 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) XLogRecPtr src_restart_lsn; bool src_islogical; bool temporary; + int inactive_timeout; /* in seconds */ char *plugin; Datum values[2]; bool nulls[2]; @@ -776,6 +794,7 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) src_restart_lsn = first_slot_contents.data.restart_lsn; temporary = (first_slot_contents.data.persistency == RS_TEMPORARY); plugin = logical_slot ? NameStr(first_slot_contents.data.plugin) : NULL; + inactive_timeout = first_slot_contents.data.inactive_timeout; /* Check type of replication slot */ if (src_islogical != logical_slot) @@ -823,6 +842,7 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) temporary, false, false, + inactive_timeout, src_restart_lsn, false); } @@ -830,6 +850,7 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) create_physical_replication_slot(NameStr(*dst_name), true, temporary, + inactive_timeout, src_restart_lsn); /* diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index bc40c454de..5315c08650 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -1221,7 +1221,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) { ReplicationSlotCreate(cmd->slotname, false, cmd->temporary ? RS_TEMPORARY : RS_PERSISTENT, - false, false, false); + false, false, false, 0); if (reserve_wal) { @@ -1252,7 +1252,7 @@ CreateReplicationSlot(CreateReplicationSlotCmd *cmd) */ ReplicationSlotCreate(cmd->slotname, true, cmd->temporary ? RS_TEMPORARY : RS_EPHEMERAL, - two_phase, failover, false); + two_phase, failover, false, 0); /* * Do options check early so that we can bail before calling the diff --git a/src/bin/pg_upgrade/info.c b/src/bin/pg_upgrade/info.c index 95c22a7200..12626987f0 100644 --- a/src/bin/pg_upgrade/info.c +++ b/src/bin/pg_upgrade/info.c @@ -676,7 +676,8 @@ get_old_cluster_logical_slot_infos(DbInfo *dbinfo, bool live_check) * removed. */ res = executeQueryOrDie(conn, "SELECT slot_name, plugin, two_phase, failover, " - "%s as caught_up, invalidation_reason IS NOT NULL as invalid " + "%s as caught_up, invalidation_reason IS NOT NULL as invalid, " + "inactive_timeout " "FROM pg_catalog.pg_replication_slots " "WHERE slot_type = 'logical' AND " "database = current_database() AND " @@ -696,6 +697,7 @@ get_old_cluster_logical_slot_infos(DbInfo *dbinfo, bool live_check) int i_failover; int i_caught_up; int i_invalid; + int i_inactive_timeout; slotinfos = (LogicalSlotInfo *) pg_malloc(sizeof(LogicalSlotInfo) * num_slots); @@ -705,6 +707,7 @@ get_old_cluster_logical_slot_infos(DbInfo *dbinfo, bool live_check) i_failover = PQfnumber(res, "failover"); i_caught_up = PQfnumber(res, "caught_up"); i_invalid = PQfnumber(res, "invalid"); + i_inactive_timeout = PQfnumber(res, "inactive_timeout"); for (int slotnum = 0; slotnum < num_slots; slotnum++) { @@ -716,6 +719,7 @@ get_old_cluster_logical_slot_infos(DbInfo *dbinfo, bool live_check) curr->failover = (strcmp(PQgetvalue(res, slotnum, i_failover), "t") == 0); curr->caught_up = (strcmp(PQgetvalue(res, slotnum, i_caught_up), "t") == 0); curr->invalid = (strcmp(PQgetvalue(res, slotnum, i_invalid), "t") == 0); + curr->inactive_timeout = atooid(PQgetvalue(res, slotnum, i_inactive_timeout)); } } diff --git a/src/bin/pg_upgrade/pg_upgrade.c b/src/bin/pg_upgrade/pg_upgrade.c index f6143b6bc4..2656056103 100644 --- a/src/bin/pg_upgrade/pg_upgrade.c +++ b/src/bin/pg_upgrade/pg_upgrade.c @@ -931,9 +931,10 @@ create_logical_replication_slots(void) appendPQExpBuffer(query, ", "); appendStringLiteralConn(query, slot_info->plugin, conn); - appendPQExpBuffer(query, ", false, %s, %s);", + appendPQExpBuffer(query, ", false, %s, %s, %d);", slot_info->two_phase ? "true" : "false", - slot_info->failover ? "true" : "false"); + slot_info->failover ? "true" : "false", + slot_info->inactive_timeout); PQclear(executeQueryOrDie(conn, "%s", query->data)); diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index 92bcb693fb..eb86d000b1 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -162,6 +162,8 @@ typedef struct bool invalid; /* if true, the slot is unusable */ bool failover; /* is the slot designated to be synced to the * physical standby? */ + int inactive_timeout; /* The amount of time in seconds the slot + * is allowed to be inactive. */ } LogicalSlotInfo; typedef struct diff --git a/src/bin/pg_upgrade/t/003_logical_slots.pl b/src/bin/pg_upgrade/t/003_logical_slots.pl index 83d71c3084..6e82d2cb7b 100644 --- a/src/bin/pg_upgrade/t/003_logical_slots.pl +++ b/src/bin/pg_upgrade/t/003_logical_slots.pl @@ -153,14 +153,17 @@ like( # TEST: Successful upgrade # Preparations for the subsequent test: -# 1. Setup logical replication (first, cleanup slots from the previous tests) +# 1. Setup logical replication (first, cleanup slots from the previous tests, +# and then create slot for this test with inactive_timeout set). my $old_connstr = $oldpub->connstr . ' dbname=postgres'; +my $inactive_timeout = 3600; $oldpub->start; $oldpub->safe_psql( 'postgres', qq[ SELECT * FROM pg_drop_replication_slot('test_slot1'); SELECT * FROM pg_drop_replication_slot('test_slot2'); + SELECT pg_create_logical_replication_slot(slot_name := 'regress_sub', plugin := 'pgoutput', inactive_timeout := $inactive_timeout); CREATE PUBLICATION regress_pub FOR ALL TABLES; ]); @@ -172,7 +175,7 @@ $sub->start; $sub->safe_psql( 'postgres', qq[ CREATE TABLE tbl (a int); - CREATE SUBSCRIPTION regress_sub CONNECTION '$old_connstr' PUBLICATION regress_pub WITH (two_phase = 'true', failover = 'true') + CREATE SUBSCRIPTION regress_sub CONNECTION '$old_connstr' PUBLICATION regress_pub WITH (slot_name = 'regress_sub', create_slot = false, two_phase = 'true', failover = 'true') ]); $sub->wait_for_subscription_sync($oldpub, 'regress_sub'); @@ -192,8 +195,8 @@ command_ok([@pg_upgrade_cmd], 'run of pg_upgrade of old cluster'); # Check that the slot 'regress_sub' has migrated to the new cluster $newpub->start; my $result = $newpub->safe_psql('postgres', - "SELECT slot_name, two_phase, failover FROM pg_replication_slots"); -is($result, qq(regress_sub|t|t), 'check the slot exists on new cluster'); + "SELECT slot_name, two_phase, failover, inactive_timeout = $inactive_timeout FROM pg_replication_slots"); +is($result, qq(regress_sub|t|t|t), 'check the slot exists on new cluster'); # Update the connection my $new_connstr = $newpub->connstr . ' dbname=postgres'; diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 2f7cfc02c6..ea4ffb509a 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11105,10 +11105,10 @@ # replication slots { oid => '3779', descr => 'create a physical replication slot', proname => 'pg_create_physical_replication_slot', provolatile => 'v', - proparallel => 'u', prorettype => 'record', proargtypes => 'name bool bool', - proallargtypes => '{name,bool,bool,name,pg_lsn}', - proargmodes => '{i,i,i,o,o}', - proargnames => '{slot_name,immediately_reserve,temporary,slot_name,lsn}', + proparallel => 'u', prorettype => 'record', proargtypes => 'name bool bool int4', + proallargtypes => '{name,bool,bool,int4,name,pg_lsn}', + proargmodes => '{i,i,i,i,o,o}', + proargnames => '{slot_name,immediately_reserve,temporary,inactive_timeout,slot_name,lsn}', prosrc => 'pg_create_physical_replication_slot' }, { oid => '4220', descr => 'copy a physical replication slot, changing temporality', @@ -11133,17 +11133,17 @@ proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f', proretset => 't', provolatile => 's', prorettype => 'record', proargtypes => '', - proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,timestamptz,bool,text,bool,bool}', - proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', - proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,inactive_since,conflicting,invalidation_reason,failover,synced}', + proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8,bool,int4,timestamptz,bool,text,bool,bool}', + proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}', + proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,safe_wal_size,two_phase,inactive_timeout,inactive_since,conflicting,invalidation_reason,failover,synced}', prosrc => 'pg_get_replication_slots' }, { oid => '3786', descr => 'set up a logical replication slot', proname => 'pg_create_logical_replication_slot', provolatile => 'v', proparallel => 'u', prorettype => 'record', - proargtypes => 'name name bool bool bool', - proallargtypes => '{name,name,bool,bool,bool,name,pg_lsn}', - proargmodes => '{i,i,i,i,i,o,o}', - proargnames => '{slot_name,plugin,temporary,twophase,failover,slot_name,lsn}', + proargtypes => 'name name bool bool bool int4', + proallargtypes => '{name,name,bool,bool,bool,int4,name,pg_lsn}', + proargmodes => '{i,i,i,i,i,i,o,o}', + proargnames => '{slot_name,plugin,temporary,twophase,failover,inactive_timeout,slot_name,lsn}', prosrc => 'pg_create_logical_replication_slot' }, { oid => '4222', descr => 'copy a logical replication slot, changing temporality and plugin', diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index 7b937d1a0c..5a812ef528 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -127,6 +127,9 @@ typedef struct ReplicationSlotPersistentData * for logical slots on the primary server. */ bool failover; + + /* The amount of time in seconds the slot is allowed to be inactive */ + int inactive_timeout; } ReplicationSlotPersistentData; /* @@ -239,7 +242,7 @@ extern void ReplicationSlotsShmemInit(void); extern void ReplicationSlotCreate(const char *name, bool db_specific, ReplicationSlotPersistency persistency, bool two_phase, bool failover, - bool synced); + bool synced, int inactive_timeout); extern void ReplicationSlotPersist(void); extern void ReplicationSlotDrop(const char *name, bool nowait); extern void ReplicationSlotDropAcquired(void); diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index a5aa2e3260..71fd2e708b 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -159,8 +159,9 @@ log_min_messages = 'debug2' $primary->append_conf('postgresql.conf', "log_min_messages = 'debug2'"); $primary->reload; +my $inactive_timeout = 3600; $primary->psql('postgres', - q{SELECT pg_create_logical_replication_slot('lsub2_slot', 'test_decoding', false, false, true);} + "SELECT pg_create_logical_replication_slot('lsub2_slot', 'test_decoding', false, false, true, $inactive_timeout);" ); $primary->psql('postgres', @@ -214,6 +215,16 @@ is( $standby1->safe_psql( "t", 'synchronized slot has got the inactive_since from the primary'); +# Confirm that the synced slot on the standby has got inactive_timeout from the +# primary. +is( $standby1->safe_psql( + 'postgres', + "SELECT inactive_timeout = $inactive_timeout FROM pg_replication_slots + WHERE slot_name = 'lsub2_slot' AND synced AND NOT temporary;" + ), + "t", + 'synced logical slot has got inactive_timeout on standby'); + ################################################## # Test that the synchronized slot will be dropped if the corresponding remote # slot on the primary server has been dropped. diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index f53c3036a6..7f3b70f598 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1473,12 +1473,13 @@ pg_replication_slots| SELECT l.slot_name, l.wal_status, l.safe_wal_size, l.two_phase, + l.inactive_timeout, l.inactive_since, l.conflicting, l.invalidation_reason, l.failover, l.synced - FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, inactive_since, conflicting, invalidation_reason, failover, synced) + FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase, inactive_timeout, inactive_since, conflicting, invalidation_reason, failover, synced) LEFT JOIN pg_database d ON ((l.datoid = d.oid))); pg_roles| SELECT pg_authid.rolname, pg_authid.rolsuper, -- 2.34.1 [application/octet-stream] v24-0004-Add-inactive_timeout-based-replication-slot-inva.patch (26.0K, ../../CALj2ACUXS0SfbHzsX8bqo+7CZhocsV52Kiu7OWGb5HVPAmJqnA@mail.gmail.com/5-v24-0004-Add-inactive_timeout-based-replication-slot-inva.patch) download | inline diff: From 62bbc6cbf5f23afe9db2a44b4ae9e513861c9306 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Tue, 26 Mar 2024 16:20:26 +0000 Subject: [PATCH v24 4/4] Add inactive_timeout based replication slot invalidation. Till now, postgres has the ability to invalidate inactive replication slots based on the amount of WAL (set via max_slot_wal_keep_size GUC) that will be needed for the slots in case they become active. However, choosing a default value for max_slot_wal_keep_size is tricky. Because the amount of WAL a customer generates, and their allocated storage will vary greatly in production, making it difficult to pin down a one-size-fits-all value. It is often easy for developers to set a timeout of say 1 or 2 or 3 days at slot level, after which the inactive slots get dropped. To achieve the above, postgres uses replication slot property last_inactive_time (the time at which the slot became inactive), and a new slot level parameter inactive_timeout and finds an opportunity to invalidate the slot based on this new mechanism. The invalidation check happens at various locations to help being as latest as possible, these locations include the following: - Whenever the slot is acquired if the slot gets invalidated due to this new mechanism, an error is emitted. - During checkpoint. Note that this new invalidation mechanism won't kick-in for the slots that are currently being synced from the primary to the standby. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com --- doc/src/sgml/func.sgml | 8 +- doc/src/sgml/system-views.sgml | 10 +- .../replication/logical/logicalfuncs.c | 2 +- src/backend/replication/logical/slotsync.c | 4 +- src/backend/replication/slot.c | 173 +++++++++++++++++- src/backend/replication/slotfuncs.c | 12 +- src/backend/replication/walsender.c | 4 +- src/backend/utils/adt/pg_upgrade_support.c | 2 +- src/bin/pg_upgrade/pg_upgrade.h | 3 +- src/include/replication/slot.h | 8 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/050_invalidate_slots.pl | 169 +++++++++++++++++ 12 files changed, 377 insertions(+), 19 deletions(-) create mode 100644 src/test/recovery/t/050_invalidate_slots.pl diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 2cc26e927a..fb1640ae12 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -28393,8 +28393,8 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset released upon any error. The optional fourth parameter, <parameter>inactive_timeout</parameter>, when set to a non-zero value, specifies the amount of time in seconds the slot is - allowed to be inactive. This function corresponds to the replication - protocol command + allowed to be inactive before getting invalidated. + This function corresponds to the replication protocol command <literal>CREATE_REPLICATION_SLOT ... PHYSICAL</literal>. </para></entry> </row> @@ -28442,8 +28442,8 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset failover. The optional sixth parameter, <parameter>inactive_timeout</parameter>, when set to a non-zero value, specifies the amount of time in seconds the slot is - allowed to be inactive. A call to this function has the same effect as - the replication protocol command + allowed to be inactive before getting invalidated. A call to this + function has the same effect as the replication protocol command <literal>CREATE_REPLICATION_SLOT ... LOGICAL</literal>. </para></entry> </row> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index a6cb13fd9d..3b09838a0b 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2528,7 +2528,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <structfield>inactive_timeout</structfield> <type>integer</type> </para> <para> - The amount of time in seconds the slot is allowed to be inactive. + The amount of time in seconds the slot is allowed to be inactive + before getting invalidated. </para></entry> </row> @@ -2582,6 +2583,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx perform logical decoding. It is set only for logical slots. </para> </listitem> + <listitem> + <para> + <literal>inactive_timeout</literal> means that the slot has been + inactive for the duration specified by slot's + <literal>inactive_timeout</literal> parameter. + </para> + </listitem> </itemizedlist> </para></entry> </row> diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c index b4dd5cce75..56fc1a45a9 100644 --- a/src/backend/replication/logical/logicalfuncs.c +++ b/src/backend/replication/logical/logicalfuncs.c @@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin else end_of_wal = GetXLogReplayRecPtr(NULL); - ReplicationSlotAcquire(NameStr(*name), true); + ReplicationSlotAcquire(NameStr(*name), true, true); PG_TRY(); { diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 91c1604d3c..b9ed12468c 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -324,7 +324,7 @@ drop_local_obsolete_slots(List *remote_slot_list) if (synced_slot) { - ReplicationSlotAcquire(NameStr(local_slot->data.name), true); + ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false); ReplicationSlotDropAcquired(); } @@ -534,7 +534,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) * InvalidatePossiblyObsoleteSlot() where it invalidates slot directly * if the slot is not acquired by other processes. */ - ReplicationSlotAcquire(remote_slot->name, true); + ReplicationSlotAcquire(remote_slot->name, true, false); Assert(slot == MyReplicationSlot); diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 5653e680a8..394428c24f 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -108,10 +108,11 @@ const char *const SlotInvalidationCauses[] = { [RS_INVAL_WAL_REMOVED] = "wal_removed", [RS_INVAL_HORIZON] = "rows_removed", [RS_INVAL_WAL_LEVEL] = "wal_level_insufficient", + [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout", }; /* Maximum number of invalidation causes */ -#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL +#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1), "array length mismatch"); @@ -159,6 +160,8 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr; static void ReplicationSlotShmemExit(int code, Datum arg); static void ReplicationSlotDropPtr(ReplicationSlot *slot); +static bool InvalidateSlotForInactiveTimeout(ReplicationSlot *slot, + bool need_locks); /* internal persistency functions */ static void RestoreSlotFromDisk(const char *name); @@ -306,7 +309,7 @@ ReplicationSlotValidateName(const char *name, int elevel) * that logical replication can be resumed after failover. * synced: True if the slot is synchronized from the primary server. * inactive_timeout: The amount of time in seconds the slot is allowed to be - * inactive. + * inactive before getting invalidated. */ void ReplicationSlotCreate(const char *name, bool db_specific, @@ -540,9 +543,14 @@ ReplicationSlotName(int index, Name name) * * An error is raised if nowait is true and the slot is currently in use. If * nowait is false, we sleep until the slot is released by the owning process. + * + * If check_for_invalidation is true, the slot is checked for invalidation + * based on its inactive_timeout parameter and an error is raised after making + * the slot ours. */ void -ReplicationSlotAcquire(const char *name, bool nowait) +ReplicationSlotAcquire(const char *name, bool nowait, + bool check_for_invalidation) { ReplicationSlot *s; int active_pid; @@ -620,6 +628,42 @@ retry: /* We made this slot active, so it's ours now. */ MyReplicationSlot = s; + /* + * Check if the given slot can be invalidated based on its + * inactive_timeout parameter. If yes, persist the invalidated state to + * disk and then error out. We do this only after making the slot ours to + * avoid anyone else acquiring it while we check for its invalidation. + */ + if (check_for_invalidation) + { + /* The slot is ours by now */ + Assert(s->active_pid == MyProcPid); + + /* + * Well, the slot is not yet ours really unless we check for the + * invalidation below. + */ + s->active_pid = 0; + if (InvalidateReplicationSlotForInactiveTimeout(s, true, true)) + { + /* + * If the slot has been invalidated, recalculate the resource + * limits. + */ + ReplicationSlotsComputeRequiredXmin(false); + ReplicationSlotsComputeRequiredLSN(); + + /* Might need it for slot clean up on error, so restore it */ + s->active_pid = MyProcPid; + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot acquire invalidated replication slot \"%s\"", + NameStr(MyReplicationSlot->data.name)), + errdetail("This slot has been invalidated because of its inactive_timeout parameter."))); + } + s->active_pid = MyProcPid; + } + /* * The call to pgstat_acquire_replslot() protects against stats for a * different slot, from before a restart or such, being present during @@ -797,7 +841,7 @@ ReplicationSlotDrop(const char *name, bool nowait) { Assert(MyReplicationSlot == NULL); - ReplicationSlotAcquire(name, nowait); + ReplicationSlotAcquire(name, nowait, false); /* * Do not allow users to drop the slots which are currently being synced @@ -820,7 +864,7 @@ ReplicationSlotAlter(const char *name, bool failover) { Assert(MyReplicationSlot == NULL); - ReplicationSlotAcquire(name, false); + ReplicationSlotAcquire(name, false, true); if (SlotIsPhysical(MyReplicationSlot)) ereport(ERROR, @@ -1522,6 +1566,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause, case RS_INVAL_WAL_LEVEL: appendStringInfoString(&err_detail, _("Logical decoding on standby requires wal_level >= logical on the primary server.")); break; + case RS_INVAL_INACTIVE_TIMEOUT: + appendStringInfoString(&err_detail, _("The slot has been inactive for more than the time specified by slot's inactive_timeout parameter.")); + break; case RS_INVAL_NONE: pg_unreachable(); } @@ -1635,6 +1682,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, if (SlotIsLogical(s)) invalidation_cause = cause; break; + case RS_INVAL_INACTIVE_TIMEOUT: + if (InvalidateReplicationSlotForInactiveTimeout(s, false, false)) + invalidation_cause = cause; + break; case RS_INVAL_NONE: pg_unreachable(); } @@ -1788,6 +1839,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, * - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given * db; dboid may be InvalidOid for shared relations * - RS_INVAL_WAL_LEVEL: is logical + * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout occurs * * NB - this runs as part of checkpoint, so avoid raising errors if possible. */ @@ -1839,6 +1891,102 @@ restart: return invalidated; } +/* + * Invalidate given slot based on its inactive_timeout parameter. + * + * Returns true if the slot has got invalidated. + * + * NB - this function also runs as part of checkpoint, so avoid raising errors + * if possible. + */ +bool +InvalidateReplicationSlotForInactiveTimeout(ReplicationSlot *slot, + bool need_locks, + bool persist_state) +{ + if (!InvalidateSlotForInactiveTimeout(slot, need_locks)) + return false; + + Assert(slot->active_pid == 0); + + SpinLockAcquire(&slot->mutex); + slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT; + + /* Make sure the invalidated state persists across server restart */ + slot->just_dirtied = true; + slot->dirty = true; + SpinLockRelease(&slot->mutex); + + if (persist_state) + { + char path[MAXPGPATH]; + + sprintf(path, "pg_replslot/%s", NameStr(slot->data.name)); + SaveSlotToPath(slot, path, ERROR); + } + + ReportSlotInvalidation(RS_INVAL_INACTIVE_TIMEOUT, false, 0, + slot->data.name, InvalidXLogRecPtr, + InvalidXLogRecPtr, InvalidTransactionId); + + return true; +} + +/* + * Helper for InvalidateReplicationSlotForInactiveTimeout + */ +static bool +InvalidateSlotForInactiveTimeout(ReplicationSlot *slot, bool need_locks) +{ + ReplicationSlotInvalidationCause inavidation_cause = RS_INVAL_NONE; + + if (slot->inactive_since == 0 || + slot->data.inactive_timeout == 0) + return false; + + /* + * Do not invalidate the slots which are currently being synced from the + * primary to the standby. + */ + if (RecoveryInProgress() && slot->data.synced) + return false; + + if (need_locks) + { + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + SpinLockAcquire(&slot->mutex); + } + + Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED)); + + /* + * Check if the slot needs to be invalidated due to inactive_timeout. We + * do this with the spinlock held to avoid race conditions -- for example + * the restart_lsn could move forward, or the slot could be dropped. + */ + if (slot->inactive_since > 0 && + slot->data.inactive_timeout > 0) + { + TimestampTz now; + + /* inactive_since is only tracked for inactive slots */ + Assert(slot->active_pid == 0); + + now = GetCurrentTimestamp(); + if (TimestampDifferenceExceeds(slot->inactive_since, now, + slot->data.inactive_timeout * 1000)) + inavidation_cause = RS_INVAL_INACTIVE_TIMEOUT; + } + + if (need_locks) + { + SpinLockRelease(&slot->mutex); + LWLockRelease(ReplicationSlotControlLock); + } + + return (inavidation_cause == RS_INVAL_INACTIVE_TIMEOUT); +} + /* * Flush all replication slots to disk. * @@ -1851,6 +1999,7 @@ void CheckPointReplicationSlots(bool is_shutdown) { int i; + bool invalidated = false; elog(DEBUG1, "performing replication slot checkpoint"); @@ -1874,6 +2023,13 @@ CheckPointReplicationSlots(bool is_shutdown) /* save the slot to disk, locking is handled in SaveSlotToPath() */ sprintf(path, "pg_replslot/%s", NameStr(s->data.name)); + /* + * Here's an opportunity to invalidate inactive replication slots + * based on timeout, so let's do it. + */ + if (InvalidateReplicationSlotForInactiveTimeout(s, true, false)) + invalidated = true; + /* * Slot's data is not flushed each time the confirmed_flush LSN is * updated as that could lead to frequent writes. However, we decide @@ -1900,6 +2056,13 @@ CheckPointReplicationSlots(bool is_shutdown) SaveSlotToPath(s, path, LOG); } LWLockRelease(ReplicationSlotAllocationLock); + + /* If the slot has been invalidated, recalculate the resource limits */ + if (invalidated) + { + ReplicationSlotsComputeRequiredXmin(false); + ReplicationSlotsComputeRequiredLSN(); + } } /* diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index 6e1d8d1f9a..4ea4db0f87 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -258,6 +258,7 @@ pg_get_replication_slots(PG_FUNCTION_ARGS) ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo; XLogRecPtr currlsn; int slotno; + bool invalidated = false; /* * We don't require any special permission to see this function's data @@ -466,6 +467,15 @@ pg_get_replication_slots(PG_FUNCTION_ARGS) LWLockRelease(ReplicationSlotControlLock); + /* + * If the slot has been invalidated, recalculate the resource limits + */ + if (invalidated) + { + ReplicationSlotsComputeRequiredXmin(false); + ReplicationSlotsComputeRequiredLSN(); + } + return (Datum) 0; } @@ -668,7 +678,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS) moveto = Min(moveto, GetXLogReplayRecPtr(NULL)); /* Acquire the slot so we "own" it */ - ReplicationSlotAcquire(NameStr(*slotname), true); + ReplicationSlotAcquire(NameStr(*slotname), true, true); /* A slot whose restart_lsn has never been reserved cannot be advanced */ if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn)) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index 5315c08650..7dda2f5a66 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd) if (cmd->slotname) { - ReplicationSlotAcquire(cmd->slotname, true); + ReplicationSlotAcquire(cmd->slotname, true, true); if (SlotIsLogical(MyReplicationSlot)) ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), @@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) Assert(!MyReplicationSlot); - ReplicationSlotAcquire(cmd->slotname, true); + ReplicationSlotAcquire(cmd->slotname, true, true); /* * Force a disconnect, so that the decoding code doesn't need to care diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index c54b08fe18..82956d58d3 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS) slot_name = PG_GETARG_NAME(0); /* Acquire the given slot */ - ReplicationSlotAcquire(NameStr(*slot_name), true); + ReplicationSlotAcquire(NameStr(*slot_name), true, false); Assert(SlotIsLogical(MyReplicationSlot)); diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index eb86d000b1..38d105c5d6 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -163,7 +163,8 @@ typedef struct bool failover; /* is the slot designated to be synced to the * physical standby? */ int inactive_timeout; /* The amount of time in seconds the slot - * is allowed to be inactive. */ + * is allowed to be inactive before + * getting invalidated. */ } LogicalSlotInfo; typedef struct diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index 5a812ef528..75b0bad083 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause RS_INVAL_HORIZON, /* wal_level insufficient for slot */ RS_INVAL_WAL_LEVEL, + /* inactive slot timeout has occurred */ + RS_INVAL_INACTIVE_TIMEOUT, } ReplicationSlotInvalidationCause; extern PGDLLIMPORT const char *const SlotInvalidationCauses[]; @@ -248,7 +250,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait); extern void ReplicationSlotDropAcquired(void); extern void ReplicationSlotAlter(const char *name, bool failover); -extern void ReplicationSlotAcquire(const char *name, bool nowait); +extern void ReplicationSlotAcquire(const char *name, bool nowait, + bool check_for_invalidation); extern void ReplicationSlotRelease(void); extern void ReplicationSlotCleanup(void); extern void ReplicationSlotSave(void); @@ -267,6 +270,9 @@ extern bool InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause XLogSegNo oldestSegno, Oid dboid, TransactionId snapshotConflictHorizon); +extern bool InvalidateReplicationSlotForInactiveTimeout(ReplicationSlot *slot, + bool need_locks, + bool persist_state); extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_lock); extern int ReplicationSlotIndex(ReplicationSlot *slot); extern bool ReplicationSlotName(int index, Name name); diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index b1eb77b1ec..708a2a3798 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -51,6 +51,7 @@ tests += { 't/040_standby_failover_slots_sync.pl', 't/041_checkpoint_at_promote.pl', 't/042_low_level_backup.pl', + 't/050_invalidate_slots.pl', ], }, } diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl new file mode 100644 index 0000000000..da04dfc7fc --- /dev/null +++ b/src/test/recovery/t/050_invalidate_slots.pl @@ -0,0 +1,169 @@ +# Copyright (c) 2024, PostgreSQL Global Development Group + +# Test for replication slots invalidation +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Utils; +use PostgreSQL::Test::Cluster; +use Test::More; +use Time::HiRes qw(usleep); + +# ============================================================================= +# Testcase start: Invalidate streaming standby's slot due to inactive_timeout + +# Initialize primary node +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init(allows_streaming => 'logical'); + +# Avoid checkpoint during the test, otherwise, the test can get unpredictable +$primary->append_conf( + 'postgresql.conf', q{ +checkpoint_timeout = 1h +autovacuum = off +}); +$primary->start; + +# Take backup +my $backup_name = 'my_backup'; +$primary->backup($backup_name); + +# Create a standby linking to the primary using the replication slot +my $standby1 = PostgreSQL::Test::Cluster->new('standby1'); +$standby1->init_from_backup($primary, $backup_name, has_streaming => 1); +$standby1->append_conf( + 'postgresql.conf', q{ +primary_slot_name = 'sb1_slot' +}); + +# Set timeout so that the slot when inactive will get invalidated after the +# timeout. +my $inactive_timeout = 5; +$primary->safe_psql( + 'postgres', qq[ + SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot', inactive_timeout := $inactive_timeout); +]); + +$standby1->start; + +# Wait until standby has replayed enough data +$primary->wait_for_catchup($standby1); + +# Check inactive_timeout is what we've set above +my $result = $primary->safe_psql( + 'postgres', qq[ + SELECT inactive_timeout = $inactive_timeout + FROM pg_replication_slots WHERE slot_name = 'sb1_slot'; +]); +is($result, "t", + 'check the inactive replication slot info for an active slot'); + +my $logstart = -s $primary->logfile; + +# Stop standby to make the replication slot on primary inactive +$standby1->stop; + +# Wait for the inactive replication slot info to be updated +$primary->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE inactive_since IS NOT NULL + AND slot_name = 'sb1_slot' + AND inactive_timeout = $inactive_timeout; +]) + or die + "Timed out while waiting for inactive replication slot info to be updated"; + +check_slots_invalidation_in_server_log($primary, 'sb1_slot', $logstart); + +# Wait for the inactive replication slots to be invalidated. +$primary->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE slot_name = 'sb1_slot' AND + invalidation_reason = 'inactive_timeout'; +]) + or die + "Timed out while waiting for inactive replication slot sb1_slot to be invalidated"; + +# Testcase end: Invalidate streaming standby's slot due to inactive_timeout +# ============================================================================= + +# ============================================================================= +# Testcase start: Invalidate logical subscriber's slot due to inactive_timeout + +my $publisher = $primary; + +# Create subscriber node +my $subscriber = PostgreSQL::Test::Cluster->new('sub'); +$subscriber->init; +$subscriber->start; + +# Create tables +$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)"); +$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)"); + +# Insert some data +$subscriber->safe_psql('postgres', + "INSERT INTO test_tbl VALUES (generate_series(1, 5));"); + +# Setup logical replication +my $publisher_connstr = $publisher->connstr . ' dbname=postgres'; +$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES"); +$publisher->safe_psql( + 'postgres', qq[ + SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput', inactive_timeout := $inactive_timeout); +]); + +$subscriber->safe_psql('postgres', + "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)" +); + +$subscriber->wait_for_subscription_sync($publisher, 'sub'); + +$result = $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl"); + +is($result, qq(5), "check initial copy was done"); + +$logstart = -s $publisher->logfile; + +# Stop subscriber to make the replication slot on publisher inactive +$subscriber->stop; + +# Wait for the inactive replication slot info to be updated +$publisher->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE inactive_since IS NOT NULL + AND slot_name = 'lsub1_slot' + AND inactive_timeout = $inactive_timeout; +]) + or die + "Timed out while waiting for inactive replication slot info to be updated"; + +check_slots_invalidation_in_server_log($publisher, 'lsub1_slot', $logstart); + +# Testcase end: Invalidate logical subscriber's slot due to inactive_timeout +# ============================================================================= + +# Check for invalidation of slot in server log. +sub check_slots_invalidation_in_server_log +{ + my ($node, $slot_name, $offset) = @_; + my $invalidated = 0; + + for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++) + { + $node->safe_psql('postgres', "CHECKPOINT"); + if ($node->log_contains( + "invalidating obsolete replication slot \"$slot_name\"", $offset)) + { + $invalidated = 1; + last; + } + usleep(100_000); + } + ok($invalidated, "check that slot $slot_name invalidation has been logged"); +} + +done_testing(); -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-26 17:52 Bertrand Drouvot <[email protected]> parent: Bharath Rupireddy <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-26 17:52 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Tue, Mar 26, 2024 at 09:59:23PM +0530, Bharath Rupireddy wrote: > On Tue, Mar 26, 2024 at 4:35 PM Bharath Rupireddy > <[email protected]> wrote: > > > > If we just sync inactive_since value for synced slots while in > > recovery from the primary, so be it. Why do we need to update it to > > the current time when the slot is being created? We don't expose slot > > creation time, no? Aren't we fine if we just sync the value from > > primary and document that fact? After the promotion, we can reset it > > to the current time so that it gets its own time. > > I'm attaching v24 patches. It implements the above idea proposed > upthread for synced slots. I've now separated > s/last_inactive_time/inactive_since and synced slots behaviour. Please > have a look. Thanks! ==== v24-0001 It's now pure mechanical changes and it looks good to me. ==== v24-0002 1 === This commit does two things: 1) Updates inactive_since for sync slots with the value received from the primary's slot. Tested it and it does that. 2 === 2) Ensures the value is set to current timestamp during the shutdown of slot sync machinery to help correctly interpret the time if the standby gets promoted without a restart. Tested it and it does that. 3 === +/* + * Reset the synced slots info such as inactive_since after shutting + * down the slot sync machinery. + */ +static void +update_synced_slots_inactive_time(void) Looks like the comment "reset" is not matching the name of the function and what it does. 4 === + /* + * We get the current time beforehand and only once to avoid + * system calls overhead while holding the lock. + */ + if (now == 0) + now = GetCurrentTimestamp(); Also +1 of having GetCurrentTimestamp() just called one time within the loop. 5 === - if (!(RecoveryInProgress() && slot->data.synced)) + if (!(InRecovery && slot->data.synced)) slot->inactive_since = GetCurrentTimestamp(); else slot->inactive_since = 0; Not related to this change but more the way RestoreSlotFromDisk() behaves here: For a sync slot on standby it will be set to zero and then later will be synchronized with the one coming from the primary. I think that's fine to have it to zero for this window of time. Now, if the standby is down and one sets sync_replication_slots to off, then inactive_since will be set to zero on the standby at startup and not synchronized (unless one triggers a manual sync). I also think that's fine but it might be worth to document this behavior (that after a standby startup inactive_since is zero until the next sync...). 6 === + print "HI $slot_name $name $inactive_since $slot_creation_time\n"; garbage? 7 === +# Capture and validate inactive_since of a given slot. +sub capture_and_validate_slot_inactive_since +{ + my ($node, $slot_name, $slot_creation_time) = @_; + my $name = $node->name; We know have capture_and_validate_slot_inactive_since at 2 places: 040_standby_failover_slots_sync.pl and 019_replslot_limit.pl. Worth to create a sub in Cluster.pm? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 03:31 shveta malik <[email protected]> parent: Bharath Rupireddy <[email protected]> 1 sibling, 0 replies; 46+ messages in thread From: shveta malik @ 2024-03-27 03:31 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Tue, Mar 26, 2024 at 9:59 PM Bharath Rupireddy <[email protected]> wrote: > > On Tue, Mar 26, 2024 at 4:35 PM Bharath Rupireddy > <[email protected]> wrote: > > > > If we just sync inactive_since value for synced slots while in > > recovery from the primary, so be it. Why do we need to update it to > > the current time when the slot is being created? We don't expose slot > > creation time, no? Aren't we fine if we just sync the value from > > primary and document that fact? After the promotion, we can reset it > > to the current time so that it gets its own time. > > I'm attaching v24 patches. It implements the above idea proposed > upthread for synced slots. I've now separated > s/last_inactive_time/inactive_since and synced slots behaviour. Please > have a look. Thanks for the patches. Few trivial comments for v24-002: 1) slot.c: + * data from the remote slot. We use InRecovery flag instead of + * RecoveryInProgress() as it always returns true even for normal + * server startup. a) Not clear what 'it' refers to. Better to use 'the latter' b) Is it better to mention the primary here: 'as the latter always returns true even on the primary server during startup'. 2) update_local_synced_slot(): - strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) + strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0 && + remote_slot->inactive_since == slot->inactive_since) When this code was written initially, the intent was to do strcmp at the end (only if absolutely needed). It will be good if we maintain the same and add new checks before strcmp. 3) update_synced_slots_inactive_time(): This assert is removed, is it intentional? Assert(s->active_pid == 0); 4) 040_standby_failover_slots_sync.pl: +# Capture the inactive_since of the slot from the standby the logical failover +# slots are synced/created on the standby. The comment is unclear, something seems missing. thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 04:38 Bharath Rupireddy <[email protected]> parent: Bertrand Drouvot <[email protected]> 0 siblings, 3 replies; 46+ messages in thread From: Bharath Rupireddy @ 2024-03-27 04:38 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot <[email protected]> wrote: > > > I'm attaching v24 patches. It implements the above idea proposed > > upthread for synced slots. > > ==== v24-0002 > > 1 === > > This commit does two things: > 1) Updates inactive_since for sync slots with the value > received from the primary's slot. > > Tested it and it does that. Thanks. I've added a test case for this. > 2 === > > 2) Ensures the value is set to current timestamp during the > shutdown of slot sync machinery to help correctly interpret the > time if the standby gets promoted without a restart. > > Tested it and it does that. Thanks. I've added a test case for this. > 3 === > > +/* > + * Reset the synced slots info such as inactive_since after shutting > + * down the slot sync machinery. > + */ > +static void > +update_synced_slots_inactive_time(void) > > Looks like the comment "reset" is not matching the name of the function and > what it does. Changed. I've also changed the function name to update_synced_slots_inactive_since to be precise on what it exactly does. > 4 === > > + /* > + * We get the current time beforehand and only once to avoid > + * system calls overhead while holding the lock. > + */ > + if (now == 0) > + now = GetCurrentTimestamp(); > > Also +1 of having GetCurrentTimestamp() just called one time within the loop. Right. > 5 === > > - if (!(RecoveryInProgress() && slot->data.synced)) > + if (!(InRecovery && slot->data.synced)) > slot->inactive_since = GetCurrentTimestamp(); > else > slot->inactive_since = 0; > > Not related to this change but more the way RestoreSlotFromDisk() behaves here: > > For a sync slot on standby it will be set to zero and then later will be > synchronized with the one coming from the primary. I think that's fine to have > it to zero for this window of time. Right. > Now, if the standby is down and one sets sync_replication_slots to off, > then inactive_since will be set to zero on the standby at startup and not > synchronized (unless one triggers a manual sync). I also think that's fine but > it might be worth to document this behavior (that after a standby startup > inactive_since is zero until the next sync...). Isn't this behaviour applicable for other slot parameters that the slot syncs from the remote slot on the primary? I've added the following note in the comments when we update inactive_since in RestoreSlotFromDisk. * Note that for synced slots after the standby starts up (i.e. after * the slots are loaded from the disk), the inactive_since will remain * zero until the next slot sync cycle. */ if (!(InRecovery && slot->data.synced)) slot->inactive_since = GetCurrentTimestamp(); else slot->inactive_since = 0; > 6 === > > + print "HI $slot_name $name $inactive_since $slot_creation_time\n"; > > garbage? Removed. > 7 === > > +# Capture and validate inactive_since of a given slot. > +sub capture_and_validate_slot_inactive_since > +{ > + my ($node, $slot_name, $slot_creation_time) = @_; > + my $name = $node->name; > > We know have capture_and_validate_slot_inactive_since at 2 places: > 040_standby_failover_slots_sync.pl and 019_replslot_limit.pl. > > Worth to create a sub in Cluster.pm? I'd second that thought for now. We might have to debate first if it's useful for all the nodes even without replication, and if yes, the naming stuff and all that. Historically, we've had such duplicated functions until recently, for instance advance_wal and log_contains. We moved them over to a common perl library Cluster.pm very recently. I'm sure we can come back later to move it to Cluster.pm. On Wed, Mar 27, 2024 at 9:02 AM shveta malik <[email protected]> wrote: > > 1) > slot.c: > + * data from the remote slot. We use InRecovery flag instead of > + * RecoveryInProgress() as it always returns true even for normal > + * server startup. > > a) Not clear what 'it' refers to. Better to use 'the latter' > b) Is it better to mention the primary here: > 'as the latter always returns true even on the primary server during startup'. Modified. > 2) > update_local_synced_slot(): > > - strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) > + strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0 && > + remote_slot->inactive_since == slot->inactive_since) > > When this code was written initially, the intent was to do strcmp at > the end (only if absolutely needed). It will be good if we maintain > the same and add new checks before strcmp. Done. > 3) > update_synced_slots_inactive_time(): > > This assert is removed, is it intentional? > Assert(s->active_pid == 0); Yes, the slot can get acquired in the corner case when someone runs pg_sync_replication_slots concurrently at this time. I'm referring to the issue reported upthread. We don't prevent one running pg_sync_replication_slots in promotion/ShutDownSlotSync phase right? Maybe we should prevent that otherwise some of the slots are synced and the standby gets promoted while others are yet-to-be-synced. > 4) > 040_standby_failover_slots_sync.pl: > > +# Capture the inactive_since of the slot from the standby the logical failover > +# slots are synced/created on the standby. > > The comment is unclear, something seems missing. Nice catch. Yes, that was wrong. I've modified it now. Please find the attached v25-0001 (made this 0001 patch now as inactive_since patch is committed) patch with the above changes. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com Attachments: [application/x-patch] v25-0001-Maintain-inactive_since-for-synced-slots-correct.patch (13.9K, ../../CALj2ACX-M+Lx6Qv4id7HtqqTKVKA_ti4whK9cgpTPHdHLi943w@mail.gmail.com/2-v25-0001-Maintain-inactive_since-for-synced-slots-correct.patch) download | inline diff: From 790f791b4200ff06cfdf55fdf1572436e9d982fd Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Wed, 27 Mar 2024 04:23:52 +0000 Subject: [PATCH v25] Maintain inactive_since for synced slots correctly. The slot's inactive_since isn't currently maintained for synced slots on the standby. The commit a11f330b55 prevents updating inactive_since with RecoveryInProgress() check in RestoreSlotFromDisk(). But, the issue is that RecoveryInProgress() always returns true in RestoreSlotFromDisk() as 'xlogctl->SharedRecoveryState' is always 'RECOVERY_STATE_CRASH' at that time. The impact of this on a promoted standby inactive_since is always NULL for all synced slots even after server restart. Above issue led us to a question as to why we can't just update inactive_since for synced slots on the standby with the value received from remote slot on the primary. This is consistent with any other slot parameter i.e. all of them are synced from the primary. This commit does two things: 1) Updates inactive_since for sync slots with the value received from the primary's slot. 2) Ensures the value is set to current timestamp during the shutdown of slot sync machinery to help correctly interpret the time if the standby gets promoted without a restart. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACWLctoiH-pSjWnEpR54q4DED6rw_BRJm5pCx86_Y01MoQ%40mail.gmail.com --- doc/src/sgml/system-views.sgml | 4 ++ src/backend/replication/logical/slotsync.c | 62 ++++++++++++++++- src/backend/replication/slot.c | 41 ++++++++---- .../t/040_standby_failover_slots_sync.pl | 66 +++++++++++++++++++ 4 files changed, 158 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 3c8dca8ca3..7713f168e7 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2530,6 +2530,10 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <para> The time since the slot has become inactive. <literal>NULL</literal> if the slot is currently being used. + Note that the slots that are being synced from a primary server + (whose <structfield>synced</structfield> field is true), will get the + <structfield>inactive_since</structfield> value from the corresponding + remote slot on the primary. </para></entry> </row> diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 30480960c5..d367c9ed3c 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -137,9 +137,12 @@ typedef struct RemoteSlot /* RS_INVAL_NONE if valid, or the reason of invalidation */ ReplicationSlotInvalidationCause invalidated; + + TimestampTz inactive_since; /* in seconds */ } RemoteSlot; static void slotsync_failure_callback(int code, Datum arg); +static void update_synced_slots_inactive_since(void); /* * If necessary, update the local synced slot's metadata based on the data @@ -167,6 +170,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) remote_slot->two_phase == slot->data.two_phase && remote_slot->failover == slot->data.failover && remote_slot->confirmed_lsn == slot->data.confirmed_flush && + remote_slot->inactive_since == slot->inactive_since && strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) return false; @@ -182,6 +186,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) slot->data.confirmed_flush = remote_slot->confirmed_lsn; slot->data.catalog_xmin = remote_slot->catalog_xmin; slot->effective_catalog_xmin = remote_slot->catalog_xmin; + slot->inactive_since = remote_slot->inactive_since; SpinLockRelease(&slot->mutex); if (xmin_changed) @@ -652,9 +657,9 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) static bool synchronize_slots(WalReceiverConn *wrconn) { -#define SLOTSYNC_COLUMN_COUNT 9 +#define SLOTSYNC_COLUMN_COUNT 10 Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, - LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID, TIMESTAMPTZOID}; WalRcvExecResult *res; TupleTableSlot *tupslot; @@ -663,7 +668,7 @@ synchronize_slots(WalReceiverConn *wrconn) bool started_tx = false; const char *query = "SELECT slot_name, plugin, confirmed_flush_lsn," " restart_lsn, catalog_xmin, two_phase, failover," - " database, invalidation_reason" + " database, invalidation_reason, inactive_since" " FROM pg_catalog.pg_replication_slots" " WHERE failover and NOT temporary"; @@ -743,6 +748,14 @@ synchronize_slots(WalReceiverConn *wrconn) remote_slot->invalidated = isnull ? RS_INVAL_NONE : GetSlotInvalidationCause(TextDatumGetCString(d)); + /* + * It is possible to get null value for inactive_since if the slot is + * active on the primary server, so handle accordingly. + */ + d = DatumGetTimestampTz(slot_getattr(tupslot, ++col, + &isnull)); + remote_slot->inactive_since = isnull ? 0 : DatumGetLSN(d); + /* Sanity check */ Assert(col == SLOTSYNC_COLUMN_COUNT); @@ -1296,6 +1309,46 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len) Assert(false); } +/* + * Update the inactive_since property for synced slots. + */ +static void +update_synced_slots_inactive_since(void) +{ + TimestampTz now = 0; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + + /* + * We get the current time beforehand and only once to avoid + * system calls overhead while holding the lock. + */ + if (now == 0) + now = GetCurrentTimestamp(); + + /* + * Set the time since the slot has become inactive after shutting + * down slot sync machinery. This helps correctly interpret the + * time if the standby gets promoted without a restart. + */ + SpinLockAcquire(&s->mutex); + s->inactive_since = now; + SpinLockRelease(&s->mutex); + } + } + + LWLockRelease(ReplicationSlotControlLock); +} + /* * Shut down the slot sync worker. */ @@ -1309,6 +1362,7 @@ ShutDownSlotSync(void) if (SlotSyncCtx->pid == InvalidPid) { SpinLockRelease(&SlotSyncCtx->mutex); + update_synced_slots_inactive_since(); return; } SpinLockRelease(&SlotSyncCtx->mutex); @@ -1341,6 +1395,8 @@ ShutDownSlotSync(void) } SpinLockRelease(&SlotSyncCtx->mutex); + + update_synced_slots_inactive_since(); } /* diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index d778c0b921..5d6882e4db 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -42,6 +42,7 @@ #include "access/transam.h" #include "access/xlog_internal.h" #include "access/xlogrecovery.h" +#include "access/xlogutils.h" #include "common/file_utils.h" #include "common/string.h" #include "miscadmin.h" @@ -655,6 +656,7 @@ ReplicationSlotRelease(void) char *slotname = NULL; /* keep compiler quiet */ bool is_logical = false; /* keep compiler quiet */ TimestampTz now = 0; + bool update_inactive_since; Assert(slot != NULL && slot->active_pid != 0); @@ -690,13 +692,19 @@ ReplicationSlotRelease(void) } /* - * Set the last inactive time after marking the slot inactive. We don't - * set it for the slots currently being synced from the primary to the - * standby because such slots are typically inactive as decoding is not - * allowed on those. + * Set the time since the slot has become inactive. + * + * Note that we don't set it for the slots currently being synced from the + * primary to the standby, because such slots typically sync the data from + * the remote slot. */ if (!(RecoveryInProgress() && slot->data.synced)) + { now = GetCurrentTimestamp(); + update_inactive_since = true; + } + else + update_inactive_since = false; if (slot->data.persistency == RS_PERSISTENT) { @@ -706,11 +714,12 @@ ReplicationSlotRelease(void) */ SpinLockAcquire(&slot->mutex); slot->active_pid = 0; - slot->inactive_since = now; + if (update_inactive_since) + slot->inactive_since = now; SpinLockRelease(&slot->mutex); ConditionVariableBroadcast(&slot->active_cv); } - else + else if (update_inactive_since) { SpinLockAcquire(&slot->mutex); slot->inactive_since = now; @@ -2369,13 +2378,21 @@ RestoreSlotFromDisk(const char *name) slot->active_pid = 0; /* - * We set the last inactive time after loading the slot from the disk - * into memory. Whoever acquires the slot i.e. makes the slot active - * will reset it. We don't set it for the slots currently being synced - * from the primary to the standby because such slots are typically - * inactive as decoding is not allowed on those. + * Set the time since the slot has become inactive after loading the + * slot from the disk into memory. Whoever acquires the slot i.e. + * makes the slot active will reset it. + * + * Note that we don't set it for the slots currently being synced from + * the primary to the standby, because such slots typically sync the + * data from the remote slot. We use InRecovery flag instead of + * RecoveryInProgress() as the latter always returns true at this time + * even on primary. + * + * Note that for synced slots after the standby starts up (i.e. after + * the slots are loaded from the disk), the inactive_since will remain + * zero until the next slot sync cycle. */ - if (!(RecoveryInProgress() && slot->data.synced)) + if (!(InRecovery && slot->data.synced)) slot->inactive_since = GetCurrentTimestamp(); else slot->inactive_since = 0; diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index f47bfd78eb..58d5177bad 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1'); $subscriber1->init; $subscriber1->start; +# Capture the time before the logical failover slot is created on the +# primary. We later call this publisher as primary anyway. +my $slot_creation_time_on_primary = $publisher->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + # Create a slot on the publisher with failover disabled $publisher->safe_psql('postgres', "SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);" @@ -174,6 +181,10 @@ $primary->poll_query_until( "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", 1); +# Capture the inactive_since of the slot from the primary +my $inactive_since_on_primary = + capture_and_validate_slot_inactive_since($primary, 'lsub1_slot', $slot_creation_time_on_primary); + # Wait for the standby to catch up so that the standby is not lagging behind # the subscriber. $primary->wait_for_replay_catchup($standby1); @@ -190,6 +201,18 @@ is( $standby1->safe_psql( "t", 'logical slots have synced as true on standby'); +# Capture the inactive_since of the synced slot on the standby +my $inactive_since_on_standby = + capture_and_validate_slot_inactive_since($standby1, 'lsub1_slot', $slot_creation_time_on_primary); + +# Synced slots on the standby must get the inactive_since from the primary. +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_primary'::timestamptz = '$inactive_since_on_standby'::timestamptz;" + ), + "t", + 'synchronized slot has got the inactive_since from the primary'); + ################################################## # Test that the synchronized slot will be dropped if the corresponding remote # slot on the primary server has been dropped. @@ -750,8 +773,28 @@ $primary->reload; $standby1->start; $primary->wait_for_replay_catchup($standby1); +# Capture the time before the standby is promoted +my $promotion_time_on_primary = $standby1->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + $standby1->promote; +# Capture the inactive_since of the synced slot after the promotion. +# Expectation here is that the slot gets its own inactive_since as part of the +# promotion. We do this check before the slot is enabled on the new primary +# below, otherwise the slot gets active setting inactive_since to NULL. +my $inactive_since_on_new_primary = + capture_and_validate_slot_inactive_since($standby1, 'lsub1_slot', $promotion_time_on_primary); + +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_new_primary'::timestamptz > '$inactive_since_on_primary'::timestamptz" + ), + "t", + 'synchronized slot has got its own inactive_since on the new primary'); + # Update subscription with the new primary's connection info my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; $subscriber1->safe_psql('postgres', @@ -773,4 +816,27 @@ is( $subscriber1->safe_psql('postgres', q{SELECT count(*) FROM tab_int;}), "20", 'data replicated from the new primary'); +# Capture and validate inactive_since of a given slot. +sub capture_and_validate_slot_inactive_since +{ + my ($node, $slot_name, $reference_time) = @_; + my $name = $node->name; + + my $inactive_since = $node->safe_psql('postgres', + qq(SELECT inactive_since FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) + ); + + # Check that the captured time is sane + is( $node->safe_psql( + 'postgres', + qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND + '$inactive_since'::timestamptz >= '$reference_time'::timestamptz;] + ), + 't', + "last inactive time for slot $slot_name is sane on node $name"); + + return $inactive_since; +} + done_testing(); -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 04:52 Amit Kapila <[email protected]> parent: Bharath Rupireddy <[email protected]> 2 siblings, 1 reply; 46+ messages in thread From: Amit Kapila @ 2024-03-27 04:52 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Mar 27, 2024 at 10:08 AM Bharath Rupireddy <[email protected]> wrote: > > On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot > <[email protected]> wrote: > > > 3) > > update_synced_slots_inactive_time(): > > > > This assert is removed, is it intentional? > > Assert(s->active_pid == 0); > > Yes, the slot can get acquired in the corner case when someone runs > pg_sync_replication_slots concurrently at this time. I'm referring to > the issue reported upthread. We don't prevent one running > pg_sync_replication_slots in promotion/ShutDownSlotSync phase right? > Maybe we should prevent that otherwise some of the slots are synced > and the standby gets promoted while others are yet-to-be-synced. > We should do something about it but that shouldn't be done in this patch. We can handle it separately and then add such an assert. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 04:54 shveta malik <[email protected]> parent: Amit Kapila <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: shveta malik @ 2024-03-27 04:54 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Bertrand Drouvot <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Wed, Mar 27, 2024 at 10:22 AM Amit Kapila <[email protected]> wrote: > > On Wed, Mar 27, 2024 at 10:08 AM Bharath Rupireddy > <[email protected]> wrote: > > > > On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot > > <[email protected]> wrote: > > > > > 3) > > > update_synced_slots_inactive_time(): > > > > > > This assert is removed, is it intentional? > > > Assert(s->active_pid == 0); > > > > Yes, the slot can get acquired in the corner case when someone runs > > pg_sync_replication_slots concurrently at this time. I'm referring to > > the issue reported upthread. We don't prevent one running > > pg_sync_replication_slots in promotion/ShutDownSlotSync phase right? > > Maybe we should prevent that otherwise some of the slots are synced > > and the standby gets promoted while others are yet-to-be-synced. > > > > We should do something about it but that shouldn't be done in this > patch. We can handle it separately and then add such an assert. Agreed. Once this patch is concluded, I can fix the slot sync shutdown issue and will also add this 'assert' back. thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 04:56 Bharath Rupireddy <[email protected]> parent: shveta malik <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Bharath Rupireddy @ 2024-03-27 04:56 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Mar 27, 2024 at 10:24 AM shveta malik <[email protected]> wrote: > > On Wed, Mar 27, 2024 at 10:22 AM Amit Kapila <[email protected]> wrote: > > > > On Wed, Mar 27, 2024 at 10:08 AM Bharath Rupireddy > > <[email protected]> wrote: > > > > > > On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot > > > <[email protected]> wrote: > > > > > > > 3) > > > > update_synced_slots_inactive_time(): > > > > > > > > This assert is removed, is it intentional? > > > > Assert(s->active_pid == 0); > > > > > > Yes, the slot can get acquired in the corner case when someone runs > > > pg_sync_replication_slots concurrently at this time. I'm referring to > > > the issue reported upthread. We don't prevent one running > > > pg_sync_replication_slots in promotion/ShutDownSlotSync phase right? > > > Maybe we should prevent that otherwise some of the slots are synced > > > and the standby gets promoted while others are yet-to-be-synced. > > > > > > > We should do something about it but that shouldn't be done in this > > patch. We can handle it separately and then add such an assert. > > Agreed. Once this patch is concluded, I can fix the slot sync shutdown > issue and will also add this 'assert' back. Agreed. Thanks. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 05:25 shveta malik <[email protected]> parent: Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: shveta malik @ 2024-03-27 05:25 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bharath Rupireddy <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Tue, Mar 26, 2024 at 6:05 PM Bertrand Drouvot <[email protected]> wrote: > > > > We can think on that later if we really need another > > field which give us sync time. > > I think that calling GetCurrentTimestamp() so frequently could be too costly, so > I'm not sure we should. Agreed. > > In my second approach, I have tried to > > avoid updating inactive_since for synced slots during sync process. We > > update that field during creation of synced slot so that > > inactive_since reflects correct info even for synced slots (rather > > than copying from primary). > > Yeah, and I think we could create a dedicated field with this information > if we feel the need. Okay. thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 05:35 Bharath Rupireddy <[email protected]> parent: Bharath Rupireddy <[email protected]> 2 siblings, 1 reply; 46+ messages in thread From: Bharath Rupireddy @ 2024-03-27 05:35 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Mar 27, 2024 at 10:08 AM Bharath Rupireddy <[email protected]> wrote: > > Please find the attached v25-0001 (made this 0001 patch now as > inactive_since patch is committed) patch with the above changes. Fixed an issue in synchronize_slots where DatumGetLSN is being used in place of DatumGetTimestampTz. Found this via CF bot member [1], not on my dev system. Please find the attached v6 patch. [1] [05:14:39.281] #7 DatumGetLSN (X=<optimized out>) at ../src/include/utils/pg_lsn.h:24 [05:14:39.281] No locals. [05:14:39.281] #8 synchronize_slots (wrconn=wrconn@entry=0x583cd170) at ../src/backend/replication/logical/slotsync.c:757 [05:14:39.281] isnull = false [05:14:39.281] remote_slot = 0x583ce1a8 [05:14:39.281] d = <optimized out> [05:14:39.281] col = 10 [05:14:39.281] slotRow = {25, 25, 3220, 3220, 28, 16, 16, 25, 25, 1184} [05:14:39.281] res = 0x583cd1b8 [05:14:39.281] tupslot = 0x583ce11c [05:14:39.281] remote_slot_list = 0x0 [05:14:39.281] some_slot_updated = false [05:14:39.281] started_tx = false [05:14:39.281] query = 0x57692bc4 "SELECT slot_name, plugin, confirmed_flush_lsn, restart_lsn, catalog_xmin, two_phase, failover, database, invalidation_reason, inactive_since FROM pg_catalog.pg_replication_slots WHERE failover and NOT"... [05:14:39.281] __func__ = "synchronize_slots" [05:14:39.281] #9 0x56ff9d1e in SyncReplicationSlots (wrconn=0x583cd170) at ../src/backend/replication/logical/slotsync.c:1504 -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com Attachments: [application/x-patch] v26-0001-Maintain-inactive_since-for-synced-slots-correct.patch (13.8K, ../../CALj2ACXEwHxzuP9imWoJKc24HcSB1izPaot7U9XGzX0qGWq3DA@mail.gmail.com/2-v26-0001-Maintain-inactive_since-for-synced-slots-correct.patch) download | inline diff: From 8c5f7d0064e0e01a2d458872ecc1d8e682ddc033 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Wed, 27 Mar 2024 05:29:18 +0000 Subject: [PATCH v26] Maintain inactive_since for synced slots correctly. The slot's inactive_since isn't currently maintained for synced slots on the standby. The commit a11f330b55 prevents updating inactive_since with RecoveryInProgress() check in RestoreSlotFromDisk(). But, the issue is that RecoveryInProgress() always returns true in RestoreSlotFromDisk() as 'xlogctl->SharedRecoveryState' is always 'RECOVERY_STATE_CRASH' at that time. The impact of this on a promoted standby inactive_since is always NULL for all synced slots even after server restart. Above issue led us to a question as to why we can't just update inactive_since for synced slots on the standby with the value received from remote slot on the primary. This is consistent with any other slot parameter i.e. all of them are synced from the primary. This commit does two things: 1) Updates inactive_since for sync slots with the value received from the primary's slot. 2) Ensures the value is set to current timestamp during the shutdown of slot sync machinery to help correctly interpret the time if the standby gets promoted without a restart. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACWLctoiH-pSjWnEpR54q4DED6rw_BRJm5pCx86_Y01MoQ%40mail.gmail.com --- doc/src/sgml/system-views.sgml | 4 ++ src/backend/replication/logical/slotsync.c | 61 ++++++++++++++++- src/backend/replication/slot.c | 41 ++++++++---- .../t/040_standby_failover_slots_sync.pl | 66 +++++++++++++++++++ 4 files changed, 157 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 3c8dca8ca3..7713f168e7 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2530,6 +2530,10 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <para> The time since the slot has become inactive. <literal>NULL</literal> if the slot is currently being used. + Note that the slots that are being synced from a primary server + (whose <structfield>synced</structfield> field is true), will get the + <structfield>inactive_since</structfield> value from the corresponding + remote slot on the primary. </para></entry> </row> diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 30480960c5..9c95a4b062 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -137,9 +137,12 @@ typedef struct RemoteSlot /* RS_INVAL_NONE if valid, or the reason of invalidation */ ReplicationSlotInvalidationCause invalidated; + + TimestampTz inactive_since; /* in seconds */ } RemoteSlot; static void slotsync_failure_callback(int code, Datum arg); +static void update_synced_slots_inactive_since(void); /* * If necessary, update the local synced slot's metadata based on the data @@ -167,6 +170,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) remote_slot->two_phase == slot->data.two_phase && remote_slot->failover == slot->data.failover && remote_slot->confirmed_lsn == slot->data.confirmed_flush && + remote_slot->inactive_since == slot->inactive_since && strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) return false; @@ -182,6 +186,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) slot->data.confirmed_flush = remote_slot->confirmed_lsn; slot->data.catalog_xmin = remote_slot->catalog_xmin; slot->effective_catalog_xmin = remote_slot->catalog_xmin; + slot->inactive_since = remote_slot->inactive_since; SpinLockRelease(&slot->mutex); if (xmin_changed) @@ -652,9 +657,9 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) static bool synchronize_slots(WalReceiverConn *wrconn) { -#define SLOTSYNC_COLUMN_COUNT 9 +#define SLOTSYNC_COLUMN_COUNT 10 Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, - LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID, TIMESTAMPTZOID}; WalRcvExecResult *res; TupleTableSlot *tupslot; @@ -663,7 +668,7 @@ synchronize_slots(WalReceiverConn *wrconn) bool started_tx = false; const char *query = "SELECT slot_name, plugin, confirmed_flush_lsn," " restart_lsn, catalog_xmin, two_phase, failover," - " database, invalidation_reason" + " database, invalidation_reason, inactive_since" " FROM pg_catalog.pg_replication_slots" " WHERE failover and NOT temporary"; @@ -743,6 +748,13 @@ synchronize_slots(WalReceiverConn *wrconn) remote_slot->invalidated = isnull ? RS_INVAL_NONE : GetSlotInvalidationCause(TextDatumGetCString(d)); + /* + * It is possible to get null value for inactive_since if the slot is + * active on the primary server, so handle accordingly. + */ + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->inactive_since = isnull ? 0 : DatumGetTimestampTz(d); + /* Sanity check */ Assert(col == SLOTSYNC_COLUMN_COUNT); @@ -1296,6 +1308,46 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len) Assert(false); } +/* + * Update the inactive_since property for synced slots. + */ +static void +update_synced_slots_inactive_since(void) +{ + TimestampTz now = 0; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + + /* + * We get the current time beforehand and only once to avoid + * system calls overhead while holding the lock. + */ + if (now == 0) + now = GetCurrentTimestamp(); + + /* + * Set the time since the slot has become inactive after shutting + * down slot sync machinery. This helps correctly interpret the + * time if the standby gets promoted without a restart. + */ + SpinLockAcquire(&s->mutex); + s->inactive_since = now; + SpinLockRelease(&s->mutex); + } + } + + LWLockRelease(ReplicationSlotControlLock); +} + /* * Shut down the slot sync worker. */ @@ -1309,6 +1361,7 @@ ShutDownSlotSync(void) if (SlotSyncCtx->pid == InvalidPid) { SpinLockRelease(&SlotSyncCtx->mutex); + update_synced_slots_inactive_since(); return; } SpinLockRelease(&SlotSyncCtx->mutex); @@ -1341,6 +1394,8 @@ ShutDownSlotSync(void) } SpinLockRelease(&SlotSyncCtx->mutex); + + update_synced_slots_inactive_since(); } /* diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index d778c0b921..5d6882e4db 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -42,6 +42,7 @@ #include "access/transam.h" #include "access/xlog_internal.h" #include "access/xlogrecovery.h" +#include "access/xlogutils.h" #include "common/file_utils.h" #include "common/string.h" #include "miscadmin.h" @@ -655,6 +656,7 @@ ReplicationSlotRelease(void) char *slotname = NULL; /* keep compiler quiet */ bool is_logical = false; /* keep compiler quiet */ TimestampTz now = 0; + bool update_inactive_since; Assert(slot != NULL && slot->active_pid != 0); @@ -690,13 +692,19 @@ ReplicationSlotRelease(void) } /* - * Set the last inactive time after marking the slot inactive. We don't - * set it for the slots currently being synced from the primary to the - * standby because such slots are typically inactive as decoding is not - * allowed on those. + * Set the time since the slot has become inactive. + * + * Note that we don't set it for the slots currently being synced from the + * primary to the standby, because such slots typically sync the data from + * the remote slot. */ if (!(RecoveryInProgress() && slot->data.synced)) + { now = GetCurrentTimestamp(); + update_inactive_since = true; + } + else + update_inactive_since = false; if (slot->data.persistency == RS_PERSISTENT) { @@ -706,11 +714,12 @@ ReplicationSlotRelease(void) */ SpinLockAcquire(&slot->mutex); slot->active_pid = 0; - slot->inactive_since = now; + if (update_inactive_since) + slot->inactive_since = now; SpinLockRelease(&slot->mutex); ConditionVariableBroadcast(&slot->active_cv); } - else + else if (update_inactive_since) { SpinLockAcquire(&slot->mutex); slot->inactive_since = now; @@ -2369,13 +2378,21 @@ RestoreSlotFromDisk(const char *name) slot->active_pid = 0; /* - * We set the last inactive time after loading the slot from the disk - * into memory. Whoever acquires the slot i.e. makes the slot active - * will reset it. We don't set it for the slots currently being synced - * from the primary to the standby because such slots are typically - * inactive as decoding is not allowed on those. + * Set the time since the slot has become inactive after loading the + * slot from the disk into memory. Whoever acquires the slot i.e. + * makes the slot active will reset it. + * + * Note that we don't set it for the slots currently being synced from + * the primary to the standby, because such slots typically sync the + * data from the remote slot. We use InRecovery flag instead of + * RecoveryInProgress() as the latter always returns true at this time + * even on primary. + * + * Note that for synced slots after the standby starts up (i.e. after + * the slots are loaded from the disk), the inactive_since will remain + * zero until the next slot sync cycle. */ - if (!(RecoveryInProgress() && slot->data.synced)) + if (!(InRecovery && slot->data.synced)) slot->inactive_since = GetCurrentTimestamp(); else slot->inactive_since = 0; diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index f47bfd78eb..58d5177bad 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1'); $subscriber1->init; $subscriber1->start; +# Capture the time before the logical failover slot is created on the +# primary. We later call this publisher as primary anyway. +my $slot_creation_time_on_primary = $publisher->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + # Create a slot on the publisher with failover disabled $publisher->safe_psql('postgres', "SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);" @@ -174,6 +181,10 @@ $primary->poll_query_until( "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", 1); +# Capture the inactive_since of the slot from the primary +my $inactive_since_on_primary = + capture_and_validate_slot_inactive_since($primary, 'lsub1_slot', $slot_creation_time_on_primary); + # Wait for the standby to catch up so that the standby is not lagging behind # the subscriber. $primary->wait_for_replay_catchup($standby1); @@ -190,6 +201,18 @@ is( $standby1->safe_psql( "t", 'logical slots have synced as true on standby'); +# Capture the inactive_since of the synced slot on the standby +my $inactive_since_on_standby = + capture_and_validate_slot_inactive_since($standby1, 'lsub1_slot', $slot_creation_time_on_primary); + +# Synced slots on the standby must get the inactive_since from the primary. +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_primary'::timestamptz = '$inactive_since_on_standby'::timestamptz;" + ), + "t", + 'synchronized slot has got the inactive_since from the primary'); + ################################################## # Test that the synchronized slot will be dropped if the corresponding remote # slot on the primary server has been dropped. @@ -750,8 +773,28 @@ $primary->reload; $standby1->start; $primary->wait_for_replay_catchup($standby1); +# Capture the time before the standby is promoted +my $promotion_time_on_primary = $standby1->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + $standby1->promote; +# Capture the inactive_since of the synced slot after the promotion. +# Expectation here is that the slot gets its own inactive_since as part of the +# promotion. We do this check before the slot is enabled on the new primary +# below, otherwise the slot gets active setting inactive_since to NULL. +my $inactive_since_on_new_primary = + capture_and_validate_slot_inactive_since($standby1, 'lsub1_slot', $promotion_time_on_primary); + +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_new_primary'::timestamptz > '$inactive_since_on_primary'::timestamptz" + ), + "t", + 'synchronized slot has got its own inactive_since on the new primary'); + # Update subscription with the new primary's connection info my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; $subscriber1->safe_psql('postgres', @@ -773,4 +816,27 @@ is( $subscriber1->safe_psql('postgres', q{SELECT count(*) FROM tab_int;}), "20", 'data replicated from the new primary'); +# Capture and validate inactive_since of a given slot. +sub capture_and_validate_slot_inactive_since +{ + my ($node, $slot_name, $reference_time) = @_; + my $name = $node->name; + + my $inactive_since = $node->safe_psql('postgres', + qq(SELECT inactive_since FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) + ); + + # Check that the captured time is sane + is( $node->safe_psql( + 'postgres', + qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND + '$inactive_since'::timestamptz >= '$reference_time'::timestamptz;] + ), + 't', + "last inactive time for slot $slot_name is sane on node $name"); + + return $inactive_since; +} + done_testing(); -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 05:48 Bertrand Drouvot <[email protected]> parent: Bharath Rupireddy <[email protected]> 2 siblings, 0 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-27 05:48 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Wed, Mar 27, 2024 at 10:08:33AM +0530, Bharath Rupireddy wrote: > On Tue, Mar 26, 2024 at 11:22 PM Bertrand Drouvot > <[email protected]> wrote: > > > > - if (!(RecoveryInProgress() && slot->data.synced)) > > + if (!(InRecovery && slot->data.synced)) > > slot->inactive_since = GetCurrentTimestamp(); > > else > > slot->inactive_since = 0; > > > > Not related to this change but more the way RestoreSlotFromDisk() behaves here: > > > > For a sync slot on standby it will be set to zero and then later will be > > synchronized with the one coming from the primary. I think that's fine to have > > it to zero for this window of time. > > Right. > > > Now, if the standby is down and one sets sync_replication_slots to off, > > then inactive_since will be set to zero on the standby at startup and not > > synchronized (unless one triggers a manual sync). I also think that's fine but > > it might be worth to document this behavior (that after a standby startup > > inactive_since is zero until the next sync...). > > Isn't this behaviour applicable for other slot parameters that the > slot syncs from the remote slot on the primary? No they are persisted on disk. If not, we'd not know where to resume the decoding from on the standby in case primary is down and/or sync is off. > I've added the following note in the comments when we update > inactive_since in RestoreSlotFromDisk. > > * Note that for synced slots after the standby starts up (i.e. after > * the slots are loaded from the disk), the inactive_since will remain > * zero until the next slot sync cycle. > */ > if (!(InRecovery && slot->data.synced)) > slot->inactive_since = GetCurrentTimestamp(); > else > slot->inactive_since = 0; I think we should add some words in the doc too and also about what the meaning of inactive_since on the standby is (as suggested by Shveta in [1]). [1]: https://www.postgresql.org/message-id/CAJpy0uDkTW%2Bt1k3oPkaipFBzZePfFNB5DmiA%3D%3DpxRGcAdpF%3DPg%40... > > 7 === > > > > +# Capture and validate inactive_since of a given slot. > > +sub capture_and_validate_slot_inactive_since > > +{ > > + my ($node, $slot_name, $slot_creation_time) = @_; > > + my $name = $node->name; > > > > We know have capture_and_validate_slot_inactive_since at 2 places: > > 040_standby_failover_slots_sync.pl and 019_replslot_limit.pl. > > > > Worth to create a sub in Cluster.pm? > > I'd second that thought for now. We might have to debate first if it's > useful for all the nodes even without replication, and if yes, the > naming stuff and all that. Historically, we've had such duplicated > functions until recently, for instance advance_wal and log_contains. > We > moved them over to a common perl library Cluster.pm very recently. I'm > sure we can come back later to move it to Cluster.pm. I thought that would be the right time not to introduce duplicated code. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 06:09 shveta malik <[email protected]> parent: Bharath Rupireddy <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: shveta malik @ 2024-03-27 06:09 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Wed, Mar 27, 2024 at 11:05 AM Bharath Rupireddy <[email protected]> wrote: > > Fixed an issue in synchronize_slots where DatumGetLSN is being used in > place of DatumGetTimestampTz. Found this via CF bot member [1], not on > my dev system. > > Please find the attached v6 patch. Thanks for the patch. Few trivial things: ---------- 1) system-views.sgml: a) "Note that the slots" --> "Note that the slots on the standbys," --it is good to mention "standbys" as synced could be true on primary as well (promoted standby) b) If you plan to add more info which Bertrand suggested, then it will be better to make a <note> section instead of using "Note" 2) commit msg: "The impact of this on a promoted standby inactive_since is always NULL for all synced slots even after server restart. " Sentence looks broken. --------- Apart from the above trivial things, v26-001 looks good to me. thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 09:25 Bharath Rupireddy <[email protected]> parent: shveta malik <[email protected]> 0 siblings, 2 replies; 46+ messages in thread From: Bharath Rupireddy @ 2024-03-27 09:25 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Mar 27, 2024 at 11:39 AM shveta malik <[email protected]> wrote: > > Thanks for the patch. Few trivial things: Thanks for reviewing. > ---------- > 1) > system-views.sgml: > > a) "Note that the slots" --> "Note that the slots on the standbys," > --it is good to mention "standbys" as synced could be true on primary > as well (promoted standby) Done. > b) If you plan to add more info which Bertrand suggested, then it will > be better to make a <note> section instead of using "Note" I added the note that Bertrand specified upthread. But, I couldn't find an instance of adding <note> ... </note> within a table. Hence with "Note that ...." statments just like any other notes in the system-views.sgml. pg_replication_slot in system-vews.sgml renders as table, so having <note> ... </note> may not be a great idea. > 2) > commit msg: > > "The impact of this > on a promoted standby inactive_since is always NULL for all > synced slots even after server restart. > " > Sentence looks broken. > --------- Reworded. > Apart from the above trivial things, v26-001 looks good to me. Please check the attached v27 patch which also has Bertrand's comment on deduplicating the TAP function. I've now moved it to Cluster.pm. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com Attachments: [application/x-patch] v27-0001-Maintain-inactive_since-for-synced-slots-correct.patch (16.4K, ../../CALj2ACXh4ur3gk1O-yT+R4sOJtKP6KSYjMYm6d4R-AtVTMhikw@mail.gmail.com/2-v27-0001-Maintain-inactive_since-for-synced-slots-correct.patch) download | inline diff: From b4f113ef1d3467383913d0f04cc672372133420d Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Wed, 27 Mar 2024 09:15:41 +0000 Subject: [PATCH v27] Maintain inactive_since for synced slots correctly. The slot's inactive_since isn't currently maintained for synced slots on the standby. The commit a11f330b55 prevents updating inactive_since with RecoveryInProgress() check in RestoreSlotFromDisk(). But, the issue is that RecoveryInProgress() always returns true in RestoreSlotFromDisk() as 'xlogctl->SharedRecoveryState' is always 'RECOVERY_STATE_CRASH' at that time. Because of this, inactive_since is always NULL on a promoted standby for all synced slots even after server restart. Above issue led us to a question as to why we can't just update inactive_since for synced slots on the standby with the value received from remote slot on the primary. This is consistent with any other slot parameter i.e. all of them are synced from the primary. This commit does two things: 1) Updates inactive_since for sync slots with the value received from the primary's slot. 2) Ensures the value is set to current timestamp during the shutdown of slot sync machinery to help correctly interpret the time if the standby gets promoted without a restart. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACWLctoiH-pSjWnEpR54q4DED6rw_BRJm5pCx86_Y01MoQ%40mail.gmail.com --- doc/src/sgml/system-views.sgml | 8 +++ src/backend/replication/logical/slotsync.c | 61 ++++++++++++++++++- src/backend/replication/slot.c | 41 +++++++++---- src/test/perl/PostgreSQL/Test/Cluster.pm | 34 +++++++++++ src/test/recovery/t/019_replslot_limit.pl | 26 +------- .../t/040_standby_failover_slots_sync.pl | 43 +++++++++++++ 6 files changed, 174 insertions(+), 39 deletions(-) diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 3c8dca8ca3..07f6d177e8 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2530,6 +2530,14 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <para> The time since the slot has become inactive. <literal>NULL</literal> if the slot is currently being used. + Note that the slots on the standbys that are being synced from a + primary server (whose <structfield>synced</structfield> field is + <literal>true</literal>), will get the + <structfield>inactive_since</structfield> value from the + corresponding remote slot on the primary. Also, note that for the + synced slots on the standby, after the standby starts up (i.e. after + the slots are loaded from the disk), the inactive_since will remain + zero until the next slot sync cycle. </para></entry> </row> diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 30480960c5..9c95a4b062 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -137,9 +137,12 @@ typedef struct RemoteSlot /* RS_INVAL_NONE if valid, or the reason of invalidation */ ReplicationSlotInvalidationCause invalidated; + + TimestampTz inactive_since; /* in seconds */ } RemoteSlot; static void slotsync_failure_callback(int code, Datum arg); +static void update_synced_slots_inactive_since(void); /* * If necessary, update the local synced slot's metadata based on the data @@ -167,6 +170,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) remote_slot->two_phase == slot->data.two_phase && remote_slot->failover == slot->data.failover && remote_slot->confirmed_lsn == slot->data.confirmed_flush && + remote_slot->inactive_since == slot->inactive_since && strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) return false; @@ -182,6 +186,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) slot->data.confirmed_flush = remote_slot->confirmed_lsn; slot->data.catalog_xmin = remote_slot->catalog_xmin; slot->effective_catalog_xmin = remote_slot->catalog_xmin; + slot->inactive_since = remote_slot->inactive_since; SpinLockRelease(&slot->mutex); if (xmin_changed) @@ -652,9 +657,9 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) static bool synchronize_slots(WalReceiverConn *wrconn) { -#define SLOTSYNC_COLUMN_COUNT 9 +#define SLOTSYNC_COLUMN_COUNT 10 Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, - LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID, TIMESTAMPTZOID}; WalRcvExecResult *res; TupleTableSlot *tupslot; @@ -663,7 +668,7 @@ synchronize_slots(WalReceiverConn *wrconn) bool started_tx = false; const char *query = "SELECT slot_name, plugin, confirmed_flush_lsn," " restart_lsn, catalog_xmin, two_phase, failover," - " database, invalidation_reason" + " database, invalidation_reason, inactive_since" " FROM pg_catalog.pg_replication_slots" " WHERE failover and NOT temporary"; @@ -743,6 +748,13 @@ synchronize_slots(WalReceiverConn *wrconn) remote_slot->invalidated = isnull ? RS_INVAL_NONE : GetSlotInvalidationCause(TextDatumGetCString(d)); + /* + * It is possible to get null value for inactive_since if the slot is + * active on the primary server, so handle accordingly. + */ + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->inactive_since = isnull ? 0 : DatumGetTimestampTz(d); + /* Sanity check */ Assert(col == SLOTSYNC_COLUMN_COUNT); @@ -1296,6 +1308,46 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len) Assert(false); } +/* + * Update the inactive_since property for synced slots. + */ +static void +update_synced_slots_inactive_since(void) +{ + TimestampTz now = 0; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + + /* + * We get the current time beforehand and only once to avoid + * system calls overhead while holding the lock. + */ + if (now == 0) + now = GetCurrentTimestamp(); + + /* + * Set the time since the slot has become inactive after shutting + * down slot sync machinery. This helps correctly interpret the + * time if the standby gets promoted without a restart. + */ + SpinLockAcquire(&s->mutex); + s->inactive_since = now; + SpinLockRelease(&s->mutex); + } + } + + LWLockRelease(ReplicationSlotControlLock); +} + /* * Shut down the slot sync worker. */ @@ -1309,6 +1361,7 @@ ShutDownSlotSync(void) if (SlotSyncCtx->pid == InvalidPid) { SpinLockRelease(&SlotSyncCtx->mutex); + update_synced_slots_inactive_since(); return; } SpinLockRelease(&SlotSyncCtx->mutex); @@ -1341,6 +1394,8 @@ ShutDownSlotSync(void) } SpinLockRelease(&SlotSyncCtx->mutex); + + update_synced_slots_inactive_since(); } /* diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index d778c0b921..5d6882e4db 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -42,6 +42,7 @@ #include "access/transam.h" #include "access/xlog_internal.h" #include "access/xlogrecovery.h" +#include "access/xlogutils.h" #include "common/file_utils.h" #include "common/string.h" #include "miscadmin.h" @@ -655,6 +656,7 @@ ReplicationSlotRelease(void) char *slotname = NULL; /* keep compiler quiet */ bool is_logical = false; /* keep compiler quiet */ TimestampTz now = 0; + bool update_inactive_since; Assert(slot != NULL && slot->active_pid != 0); @@ -690,13 +692,19 @@ ReplicationSlotRelease(void) } /* - * Set the last inactive time after marking the slot inactive. We don't - * set it for the slots currently being synced from the primary to the - * standby because such slots are typically inactive as decoding is not - * allowed on those. + * Set the time since the slot has become inactive. + * + * Note that we don't set it for the slots currently being synced from the + * primary to the standby, because such slots typically sync the data from + * the remote slot. */ if (!(RecoveryInProgress() && slot->data.synced)) + { now = GetCurrentTimestamp(); + update_inactive_since = true; + } + else + update_inactive_since = false; if (slot->data.persistency == RS_PERSISTENT) { @@ -706,11 +714,12 @@ ReplicationSlotRelease(void) */ SpinLockAcquire(&slot->mutex); slot->active_pid = 0; - slot->inactive_since = now; + if (update_inactive_since) + slot->inactive_since = now; SpinLockRelease(&slot->mutex); ConditionVariableBroadcast(&slot->active_cv); } - else + else if (update_inactive_since) { SpinLockAcquire(&slot->mutex); slot->inactive_since = now; @@ -2369,13 +2378,21 @@ RestoreSlotFromDisk(const char *name) slot->active_pid = 0; /* - * We set the last inactive time after loading the slot from the disk - * into memory. Whoever acquires the slot i.e. makes the slot active - * will reset it. We don't set it for the slots currently being synced - * from the primary to the standby because such slots are typically - * inactive as decoding is not allowed on those. + * Set the time since the slot has become inactive after loading the + * slot from the disk into memory. Whoever acquires the slot i.e. + * makes the slot active will reset it. + * + * Note that we don't set it for the slots currently being synced from + * the primary to the standby, because such slots typically sync the + * data from the remote slot. We use InRecovery flag instead of + * RecoveryInProgress() as the latter always returns true at this time + * even on primary. + * + * Note that for synced slots after the standby starts up (i.e. after + * the slots are loaded from the disk), the inactive_since will remain + * zero until the next slot sync cycle. */ - if (!(RecoveryInProgress() && slot->data.synced)) + if (!(InRecovery && slot->data.synced)) slot->inactive_since = GetCurrentTimestamp(); else slot->inactive_since = 0; diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index b08296605c..221fe93a8b 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -3276,6 +3276,40 @@ sub create_logical_slot_on_standby =pod +=item $node->create_logical_slot_on_standby(self, primary, slot_name, dbname) + +Get inactive_since column value for a given replication slot validating it +against given reference time. + +=cut + +sub get_slot_inactive_since_value +{ + my ($self, $slot_name, $reference_time) = @_; + my $name = $self->name; + + my $inactive_since = $self->safe_psql('postgres', + qq(SELECT inactive_since FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) + ); + + # Check that the captured time is sane + if (defined $reference_time) + { + is($self->safe_psql('postgres', + qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND + '$inactive_since'::timestamptz >= '$reference_time'::timestamptz;] + ), + 't', + "last inactive time for slot $slot_name is valid on node $name") + or die "could not validate captured inactive_since for slot $slot_name"; + } + + return $inactive_since; +} + +=pod + =item $node->advance_wal(num) Advance WAL of node by given number of segments. diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl index 3b9a306a8b..c8e5e5054d 100644 --- a/src/test/recovery/t/019_replslot_limit.pl +++ b/src/test/recovery/t/019_replslot_limit.pl @@ -443,7 +443,7 @@ $primary4->safe_psql( # Get inactive_since value after the slot's creation. Note that the slot is # still inactive till it's used by the standby below. my $inactive_since = - capture_and_validate_slot_inactive_since($primary4, $sb4_slot, $slot_creation_time); + $primary4->get_slot_inactive_since_value($sb4_slot, $slot_creation_time); $standby4->start; @@ -502,7 +502,7 @@ $publisher4->safe_psql('postgres', # Get inactive_since value after the slot's creation. Note that the slot is # still inactive till it's used by the subscriber below. $inactive_since = - capture_and_validate_slot_inactive_since($publisher4, $lsub4_slot, $slot_creation_time); + $publisher4->get_slot_inactive_since_value($lsub4_slot, $slot_creation_time); $subscriber4->start; $subscriber4->safe_psql('postgres', @@ -540,26 +540,4 @@ is( $publisher4->safe_psql( $publisher4->stop; $subscriber4->stop; -# Capture and validate inactive_since of a given slot. -sub capture_and_validate_slot_inactive_since -{ - my ($node, $slot_name, $slot_creation_time) = @_; - - my $inactive_since = $node->safe_psql('postgres', - qq(SELECT inactive_since FROM pg_replication_slots - WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) - ); - - # Check that the captured time is sane - is( $node->safe_psql( - 'postgres', - qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND - '$inactive_since'::timestamptz >= '$slot_creation_time'::timestamptz;] - ), - 't', - "last inactive time for an active slot $slot_name is sane"); - - return $inactive_since; -} - done_testing(); diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index f47bfd78eb..33e3a8dcf0 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1'); $subscriber1->init; $subscriber1->start; +# Capture the time before the logical failover slot is created on the +# primary. We later call this publisher as primary anyway. +my $slot_creation_time_on_primary = $publisher->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + # Create a slot on the publisher with failover disabled $publisher->safe_psql('postgres', "SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);" @@ -174,6 +181,10 @@ $primary->poll_query_until( "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", 1); +# Capture the inactive_since of the slot from the primary +my $inactive_since_on_primary = + $primary->get_slot_inactive_since_value('lsub1_slot', $slot_creation_time_on_primary); + # Wait for the standby to catch up so that the standby is not lagging behind # the subscriber. $primary->wait_for_replay_catchup($standby1); @@ -190,6 +201,18 @@ is( $standby1->safe_psql( "t", 'logical slots have synced as true on standby'); +# Capture the inactive_since of the synced slot on the standby +my $inactive_since_on_standby = + $standby1->get_slot_inactive_since_value('lsub1_slot', $slot_creation_time_on_primary); + +# Synced slots on the standby must get the inactive_since from the primary. +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_primary'::timestamptz = '$inactive_since_on_standby'::timestamptz;" + ), + "t", + 'synchronized slot has got the inactive_since from the primary'); + ################################################## # Test that the synchronized slot will be dropped if the corresponding remote # slot on the primary server has been dropped. @@ -750,8 +773,28 @@ $primary->reload; $standby1->start; $primary->wait_for_replay_catchup($standby1); +# Capture the time before the standby is promoted +my $promotion_time_on_primary = $standby1->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + $standby1->promote; +# Capture the inactive_since of the synced slot after the promotion. +# Expectation here is that the slot gets its own inactive_since as part of the +# promotion. We do this check before the slot is enabled on the new primary +# below, otherwise the slot gets active setting inactive_since to NULL. +my $inactive_since_on_new_primary = + $standby1->get_slot_inactive_since_value('lsub1_slot', $promotion_time_on_primary); + +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_new_primary'::timestamptz > '$inactive_since_on_primary'::timestamptz" + ), + "t", + 'synchronized slot has got its own inactive_since on the new primary'); + # Update subscription with the new primary's connection info my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; $subscriber1->safe_psql('postgres', -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 10:11 Bertrand Drouvot <[email protected]> parent: Bharath Rupireddy <[email protected]> 1 sibling, 0 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-27 10:11 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Wed, Mar 27, 2024 at 02:55:17PM +0530, Bharath Rupireddy wrote: > Please check the attached v27 patch which also has Bertrand's comment > on deduplicating the TAP function. I've now moved it to Cluster.pm. Thanks! 1 === + Note that the slots on the standbys that are being synced from a + primary server (whose <structfield>synced</structfield> field is + <literal>true</literal>), will get the + <structfield>inactive_since</structfield> value from the + corresponding remote slot on the primary. Also, note that for the + synced slots on the standby, after the standby starts up (i.e. after + the slots are loaded from the disk), the inactive_since will remain + zero until the next slot sync cycle. Not sure we should mention the "(i.e. after the slots are loaded from the disk)" and also "cycle" (as that does not sound right in case of manual sync). My proposal (in text) but feel free to reword it: Note that the slots on the standbys that are being synced from a primary server (whose synced field is true), will get the inactive_since value from the corresponding remote slot on the primary. Also, after the standby starts up, the inactive_since (for such synced slots) will remain zero until the next synchronization. 2 === +=item $node->create_logical_slot_on_standby(self, primary, slot_name, dbname) get_slot_inactive_since_value instead? 3 === +against given reference time. s/given reference/optional given reference/? Apart from the above, LGTM. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 10:13 shveta malik <[email protected]> parent: Bharath Rupireddy <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: shveta malik @ 2024-03-27 10:13 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Wed, Mar 27, 2024 at 2:55 PM Bharath Rupireddy <[email protected]> wrote: > > On Wed, Mar 27, 2024 at 11:39 AM shveta malik <[email protected]> wrote: > > > > Thanks for the patch. Few trivial things: > > Thanks for reviewing. > > > ---------- > > 1) > > system-views.sgml: > > > > a) "Note that the slots" --> "Note that the slots on the standbys," > > --it is good to mention "standbys" as synced could be true on primary > > as well (promoted standby) > > Done. > > > b) If you plan to add more info which Bertrand suggested, then it will > > be better to make a <note> section instead of using "Note" > > I added the note that Bertrand specified upthread. But, I couldn't > find an instance of adding <note> ... </note> within a table. Hence > with "Note that ...." statments just like any other notes in the > system-views.sgml. pg_replication_slot in system-vews.sgml renders as > table, so having <note> ... </note> may not be a great idea. > > > 2) > > commit msg: > > > > "The impact of this > > on a promoted standby inactive_since is always NULL for all > > synced slots even after server restart. > > " > > Sentence looks broken. > > --------- > > Reworded. > > > Apart from the above trivial things, v26-001 looks good to me. > > Please check the attached v27 patch which also has Bertrand's comment > on deduplicating the TAP function. I've now moved it to Cluster.pm. > Thanks for the patch. Regarding doc, I have few comments. + Note that the slots on the standbys that are being synced from a + primary server (whose <structfield>synced</structfield> field is + <literal>true</literal>), will get the + <structfield>inactive_since</structfield> value from the + corresponding remote slot on the primary. Also, note that for the + synced slots on the standby, after the standby starts up (i.e. after + the slots are loaded from the disk), the inactive_since will remain + zero until the next slot sync cycle. a) "inactive_since will remain zero" Since it is user exposed info and the user finds it NULL in pg_replication_slots, shall we mention NULL instead of 0? b) Since we are referring to the sync cycle here, I feel it will be good to give a link to that page. + zero until the next slot sync cycle (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/> for + slot synchronization details). thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 12:25 Bharath Rupireddy <[email protected]> parent: shveta malik <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Bharath Rupireddy @ 2024-03-27 12:25 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Mar 27, 2024 at 3:42 PM Bertrand Drouvot <[email protected]> wrote: > > 1 === > > My proposal (in text) but feel free to reword it: > > Note that the slots on the standbys that are being synced from a > primary server (whose synced field is true), will get the inactive_since value > from the corresponding remote slot on the primary. Also, after the standby starts > up, the inactive_since (for such synced slots) will remain zero until the next > synchronization. WFM. > 2 === > > +=item $node->create_logical_slot_on_standby(self, primary, slot_name, dbname) > > get_slot_inactive_since_value instead? Ugh. Changed. > 3 === > > +against given reference time. > > s/given reference/optional given reference/? Done. > Apart from the above, LGTM. Thanks for reviewing. On Wed, Mar 27, 2024 at 3:43 PM shveta malik <[email protected]> wrote: > > Thanks for the patch. Regarding doc, I have few comments. Thanks for reviewing. > a) "inactive_since will remain zero" > Since it is user exposed info and the user finds it NULL in > pg_replication_slots, shall we mention NULL instead of 0? Right. Changed. > b) Since we are referring to the sync cycle here, I feel it will be > good to give a link to that page. > + zero until the next slot sync cycle (see > + <xref linkend="logicaldecoding-replication-slots-synchronization"/> for > + slot synchronization details). WFM. Please see the attached v28 patch. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com Attachments: [application/octet-stream] v28-0001-Maintain-inactive_since-for-synced-slots-correct.patch (16.5K, ../../CALj2ACVPaiTHokRzV135UycNozKsLwa-jvcjX7G7rFbHYJLjaA@mail.gmail.com/2-v28-0001-Maintain-inactive_since-for-synced-slots-correct.patch) download | inline diff: From 769ed57bad0cb154e9889c48141ec97ee18eb790 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Wed, 27 Mar 2024 11:57:13 +0000 Subject: [PATCH v28] Maintain inactive_since for synced slots correctly. The slot's inactive_since isn't currently maintained for synced slots on the standby. The commit a11f330b55 prevents updating inactive_since with RecoveryInProgress() check in RestoreSlotFromDisk(). But, the issue is that RecoveryInProgress() always returns true in RestoreSlotFromDisk() as 'xlogctl->SharedRecoveryState' is always 'RECOVERY_STATE_CRASH' at that time. Because of this, inactive_since is always NULL on a promoted standby for all synced slots even after server restart. Above issue led us to a question as to why we can't just update inactive_since for synced slots on the standby with the value received from remote slot on the primary. This is consistent with any other slot parameter i.e. all of them are synced from the primary. This commit does two things: 1) Updates inactive_since for sync slots with the value received from the primary's slot. 2) Ensures the value is set to current timestamp during the shutdown of slot sync machinery to help correctly interpret the time if the standby gets promoted without a restart. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACWLctoiH-pSjWnEpR54q4DED6rw_BRJm5pCx86_Y01MoQ%40mail.gmail.com --- doc/src/sgml/system-views.sgml | 9 +++ src/backend/replication/logical/slotsync.c | 61 ++++++++++++++++++- src/backend/replication/slot.c | 41 +++++++++---- src/test/perl/PostgreSQL/Test/Cluster.pm | 34 +++++++++++ src/test/recovery/t/019_replslot_limit.pl | 26 +------- .../t/040_standby_failover_slots_sync.pl | 43 +++++++++++++ 6 files changed, 175 insertions(+), 39 deletions(-) diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 3c8dca8ca3..c8d97ab375 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2530,6 +2530,15 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <para> The time since the slot has become inactive. <literal>NULL</literal> if the slot is currently being used. + Note that the slots on the standbys that are being synced from a + primary server (whose <structfield>synced</structfield> field is + <literal>true</literal>), will get the + <structfield>inactive_since</structfield> value from the + corresponding remote slot on the primary. Also, after the standby + starts up, the <structfield>inactive_since</structfield> value + (for such synced slots) will remain <literal>NULL</literal> until + the next synchronization (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>). </para></entry> </row> diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 30480960c5..9c95a4b062 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -137,9 +137,12 @@ typedef struct RemoteSlot /* RS_INVAL_NONE if valid, or the reason of invalidation */ ReplicationSlotInvalidationCause invalidated; + + TimestampTz inactive_since; /* in seconds */ } RemoteSlot; static void slotsync_failure_callback(int code, Datum arg); +static void update_synced_slots_inactive_since(void); /* * If necessary, update the local synced slot's metadata based on the data @@ -167,6 +170,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) remote_slot->two_phase == slot->data.two_phase && remote_slot->failover == slot->data.failover && remote_slot->confirmed_lsn == slot->data.confirmed_flush && + remote_slot->inactive_since == slot->inactive_since && strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) return false; @@ -182,6 +186,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) slot->data.confirmed_flush = remote_slot->confirmed_lsn; slot->data.catalog_xmin = remote_slot->catalog_xmin; slot->effective_catalog_xmin = remote_slot->catalog_xmin; + slot->inactive_since = remote_slot->inactive_since; SpinLockRelease(&slot->mutex); if (xmin_changed) @@ -652,9 +657,9 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) static bool synchronize_slots(WalReceiverConn *wrconn) { -#define SLOTSYNC_COLUMN_COUNT 9 +#define SLOTSYNC_COLUMN_COUNT 10 Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, - LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID, TIMESTAMPTZOID}; WalRcvExecResult *res; TupleTableSlot *tupslot; @@ -663,7 +668,7 @@ synchronize_slots(WalReceiverConn *wrconn) bool started_tx = false; const char *query = "SELECT slot_name, plugin, confirmed_flush_lsn," " restart_lsn, catalog_xmin, two_phase, failover," - " database, invalidation_reason" + " database, invalidation_reason, inactive_since" " FROM pg_catalog.pg_replication_slots" " WHERE failover and NOT temporary"; @@ -743,6 +748,13 @@ synchronize_slots(WalReceiverConn *wrconn) remote_slot->invalidated = isnull ? RS_INVAL_NONE : GetSlotInvalidationCause(TextDatumGetCString(d)); + /* + * It is possible to get null value for inactive_since if the slot is + * active on the primary server, so handle accordingly. + */ + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->inactive_since = isnull ? 0 : DatumGetTimestampTz(d); + /* Sanity check */ Assert(col == SLOTSYNC_COLUMN_COUNT); @@ -1296,6 +1308,46 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len) Assert(false); } +/* + * Update the inactive_since property for synced slots. + */ +static void +update_synced_slots_inactive_since(void) +{ + TimestampTz now = 0; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + + /* + * We get the current time beforehand and only once to avoid + * system calls overhead while holding the lock. + */ + if (now == 0) + now = GetCurrentTimestamp(); + + /* + * Set the time since the slot has become inactive after shutting + * down slot sync machinery. This helps correctly interpret the + * time if the standby gets promoted without a restart. + */ + SpinLockAcquire(&s->mutex); + s->inactive_since = now; + SpinLockRelease(&s->mutex); + } + } + + LWLockRelease(ReplicationSlotControlLock); +} + /* * Shut down the slot sync worker. */ @@ -1309,6 +1361,7 @@ ShutDownSlotSync(void) if (SlotSyncCtx->pid == InvalidPid) { SpinLockRelease(&SlotSyncCtx->mutex); + update_synced_slots_inactive_since(); return; } SpinLockRelease(&SlotSyncCtx->mutex); @@ -1341,6 +1394,8 @@ ShutDownSlotSync(void) } SpinLockRelease(&SlotSyncCtx->mutex); + + update_synced_slots_inactive_since(); } /* diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index d778c0b921..5d6882e4db 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -42,6 +42,7 @@ #include "access/transam.h" #include "access/xlog_internal.h" #include "access/xlogrecovery.h" +#include "access/xlogutils.h" #include "common/file_utils.h" #include "common/string.h" #include "miscadmin.h" @@ -655,6 +656,7 @@ ReplicationSlotRelease(void) char *slotname = NULL; /* keep compiler quiet */ bool is_logical = false; /* keep compiler quiet */ TimestampTz now = 0; + bool update_inactive_since; Assert(slot != NULL && slot->active_pid != 0); @@ -690,13 +692,19 @@ ReplicationSlotRelease(void) } /* - * Set the last inactive time after marking the slot inactive. We don't - * set it for the slots currently being synced from the primary to the - * standby because such slots are typically inactive as decoding is not - * allowed on those. + * Set the time since the slot has become inactive. + * + * Note that we don't set it for the slots currently being synced from the + * primary to the standby, because such slots typically sync the data from + * the remote slot. */ if (!(RecoveryInProgress() && slot->data.synced)) + { now = GetCurrentTimestamp(); + update_inactive_since = true; + } + else + update_inactive_since = false; if (slot->data.persistency == RS_PERSISTENT) { @@ -706,11 +714,12 @@ ReplicationSlotRelease(void) */ SpinLockAcquire(&slot->mutex); slot->active_pid = 0; - slot->inactive_since = now; + if (update_inactive_since) + slot->inactive_since = now; SpinLockRelease(&slot->mutex); ConditionVariableBroadcast(&slot->active_cv); } - else + else if (update_inactive_since) { SpinLockAcquire(&slot->mutex); slot->inactive_since = now; @@ -2369,13 +2378,21 @@ RestoreSlotFromDisk(const char *name) slot->active_pid = 0; /* - * We set the last inactive time after loading the slot from the disk - * into memory. Whoever acquires the slot i.e. makes the slot active - * will reset it. We don't set it for the slots currently being synced - * from the primary to the standby because such slots are typically - * inactive as decoding is not allowed on those. + * Set the time since the slot has become inactive after loading the + * slot from the disk into memory. Whoever acquires the slot i.e. + * makes the slot active will reset it. + * + * Note that we don't set it for the slots currently being synced from + * the primary to the standby, because such slots typically sync the + * data from the remote slot. We use InRecovery flag instead of + * RecoveryInProgress() as the latter always returns true at this time + * even on primary. + * + * Note that for synced slots after the standby starts up (i.e. after + * the slots are loaded from the disk), the inactive_since will remain + * zero until the next slot sync cycle. */ - if (!(RecoveryInProgress() && slot->data.synced)) + if (!(InRecovery && slot->data.synced)) slot->inactive_since = GetCurrentTimestamp(); else slot->inactive_since = 0; diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index b08296605c..236b2125e4 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -3276,6 +3276,40 @@ sub create_logical_slot_on_standby =pod +=item $node->get_slot_inactive_since_value(self, primary, slot_name, dbname) + +Get inactive_since column value for a given replication slot validating it +against optional reference time. + +=cut + +sub get_slot_inactive_since_value +{ + my ($self, $slot_name, $reference_time) = @_; + my $name = $self->name; + + my $inactive_since = $self->safe_psql('postgres', + qq(SELECT inactive_since FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) + ); + + # Check that the captured time is sane + if (defined $reference_time) + { + is($self->safe_psql('postgres', + qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND + '$inactive_since'::timestamptz >= '$reference_time'::timestamptz;] + ), + 't', + "last inactive time for slot $slot_name is valid on node $name") + or die "could not validate captured inactive_since for slot $slot_name"; + } + + return $inactive_since; +} + +=pod + =item $node->advance_wal(num) Advance WAL of node by given number of segments. diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl index 3b9a306a8b..c8e5e5054d 100644 --- a/src/test/recovery/t/019_replslot_limit.pl +++ b/src/test/recovery/t/019_replslot_limit.pl @@ -443,7 +443,7 @@ $primary4->safe_psql( # Get inactive_since value after the slot's creation. Note that the slot is # still inactive till it's used by the standby below. my $inactive_since = - capture_and_validate_slot_inactive_since($primary4, $sb4_slot, $slot_creation_time); + $primary4->get_slot_inactive_since_value($sb4_slot, $slot_creation_time); $standby4->start; @@ -502,7 +502,7 @@ $publisher4->safe_psql('postgres', # Get inactive_since value after the slot's creation. Note that the slot is # still inactive till it's used by the subscriber below. $inactive_since = - capture_and_validate_slot_inactive_since($publisher4, $lsub4_slot, $slot_creation_time); + $publisher4->get_slot_inactive_since_value($lsub4_slot, $slot_creation_time); $subscriber4->start; $subscriber4->safe_psql('postgres', @@ -540,26 +540,4 @@ is( $publisher4->safe_psql( $publisher4->stop; $subscriber4->stop; -# Capture and validate inactive_since of a given slot. -sub capture_and_validate_slot_inactive_since -{ - my ($node, $slot_name, $slot_creation_time) = @_; - - my $inactive_since = $node->safe_psql('postgres', - qq(SELECT inactive_since FROM pg_replication_slots - WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) - ); - - # Check that the captured time is sane - is( $node->safe_psql( - 'postgres', - qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND - '$inactive_since'::timestamptz >= '$slot_creation_time'::timestamptz;] - ), - 't', - "last inactive time for an active slot $slot_name is sane"); - - return $inactive_since; -} - done_testing(); diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index f47bfd78eb..33e3a8dcf0 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1'); $subscriber1->init; $subscriber1->start; +# Capture the time before the logical failover slot is created on the +# primary. We later call this publisher as primary anyway. +my $slot_creation_time_on_primary = $publisher->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + # Create a slot on the publisher with failover disabled $publisher->safe_psql('postgres', "SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);" @@ -174,6 +181,10 @@ $primary->poll_query_until( "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", 1); +# Capture the inactive_since of the slot from the primary +my $inactive_since_on_primary = + $primary->get_slot_inactive_since_value('lsub1_slot', $slot_creation_time_on_primary); + # Wait for the standby to catch up so that the standby is not lagging behind # the subscriber. $primary->wait_for_replay_catchup($standby1); @@ -190,6 +201,18 @@ is( $standby1->safe_psql( "t", 'logical slots have synced as true on standby'); +# Capture the inactive_since of the synced slot on the standby +my $inactive_since_on_standby = + $standby1->get_slot_inactive_since_value('lsub1_slot', $slot_creation_time_on_primary); + +# Synced slots on the standby must get the inactive_since from the primary. +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_primary'::timestamptz = '$inactive_since_on_standby'::timestamptz;" + ), + "t", + 'synchronized slot has got the inactive_since from the primary'); + ################################################## # Test that the synchronized slot will be dropped if the corresponding remote # slot on the primary server has been dropped. @@ -750,8 +773,28 @@ $primary->reload; $standby1->start; $primary->wait_for_replay_catchup($standby1); +# Capture the time before the standby is promoted +my $promotion_time_on_primary = $standby1->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + $standby1->promote; +# Capture the inactive_since of the synced slot after the promotion. +# Expectation here is that the slot gets its own inactive_since as part of the +# promotion. We do this check before the slot is enabled on the new primary +# below, otherwise the slot gets active setting inactive_since to NULL. +my $inactive_since_on_new_primary = + $standby1->get_slot_inactive_since_value('lsub1_slot', $promotion_time_on_primary); + +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_new_primary'::timestamptz > '$inactive_since_on_primary'::timestamptz" + ), + "t", + 'synchronized slot has got its own inactive_since on the new primary'); + # Update subscription with the new primary's connection info my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; $subscriber1->safe_psql('postgres', -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 13:24 Bertrand Drouvot <[email protected]> parent: Bharath Rupireddy <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-27 13:24 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Wed, Mar 27, 2024 at 05:55:05PM +0530, Bharath Rupireddy wrote: > On Wed, Mar 27, 2024 at 3:42 PM Bertrand Drouvot > Please see the attached v28 patch. Thanks! 1 === sorry I missed it in the previous review if (!(RecoveryInProgress() && slot->data.synced)) + { now = GetCurrentTimestamp(); + update_inactive_since = true; + } + else + update_inactive_since = false; I think update_inactive_since is not needed, we could rely on (now > 0) instead. 2 === +=item $node->get_slot_inactive_since_value(self, primary, slot_name, dbname) + +Get inactive_since column value for a given replication slot validating it +against optional reference time. + +=cut + +sub get_slot_inactive_since_value +{ shouldn't be "=item $node->get_slot_inactive_since_value(self, slot_name, reference_time)" instead? Apart from the above, LGTM. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 15:30 Bharath Rupireddy <[email protected]> parent: Bertrand Drouvot <[email protected]> 0 siblings, 4 replies; 46+ messages in thread From: Bharath Rupireddy @ 2024-03-27 15:30 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Mar 27, 2024 at 6:54 PM Bertrand Drouvot <[email protected]> wrote: > > Hi, > > On Wed, Mar 27, 2024 at 05:55:05PM +0530, Bharath Rupireddy wrote: > > On Wed, Mar 27, 2024 at 3:42 PM Bertrand Drouvot > > Please see the attached v28 patch. > > Thanks! > > 1 === sorry I missed it in the previous review > > if (!(RecoveryInProgress() && slot->data.synced)) > + { > now = GetCurrentTimestamp(); > + update_inactive_since = true; > + } > + else > + update_inactive_since = false; > > I think update_inactive_since is not needed, we could rely on (now > 0) instead. Thought of using it, but, at the expense of readability. I prefer to use a variable instead. However, I changed the variable to be more meaningful to is_slot_being_synced. > 2 === > > +=item $node->get_slot_inactive_since_value(self, primary, slot_name, dbname) > + > +Get inactive_since column value for a given replication slot validating it > +against optional reference time. > + > +=cut > + > +sub get_slot_inactive_since_value > +{ > > shouldn't be "=item $node->get_slot_inactive_since_value(self, slot_name, reference_time)" > instead? Ugh. Changed. > Apart from the above, LGTM. Thanks. I'm attaching v29 patches. 0001 managing inactive_since on the standby for sync slots. 0002 implementing inactive timeout GUC based invalidation mechanism. Please have a look. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com Attachments: [application/octet-stream] v29-0001-Maintain-inactive_since-for-synced-slots-correct.patch (16.5K, ../../CALj2ACX1bdaX2gf5ttwk3buiDrgZVV=vM6S1SQKf=Sxhj1HkzQ@mail.gmail.com/2-v29-0001-Maintain-inactive_since-for-synced-slots-correct.patch) download | inline diff: From 5012d35b4e4d0631a429df7411185711cdf948fa Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Wed, 27 Mar 2024 14:21:01 +0000 Subject: [PATCH v29 1/2] Maintain inactive_since for synced slots correctly. The slot's inactive_since isn't currently maintained for synced slots on the standby. The commit a11f330b55 prevents updating inactive_since with RecoveryInProgress() check in RestoreSlotFromDisk(). But, the issue is that RecoveryInProgress() always returns true in RestoreSlotFromDisk() as 'xlogctl->SharedRecoveryState' is always 'RECOVERY_STATE_CRASH' at that time. Because of this, inactive_since is always NULL on a promoted standby for all synced slots even after server restart. Above issue led us to a question as to why we can't just update inactive_since for synced slots on the standby with the value received from remote slot on the primary. This is consistent with any other slot parameter i.e. all of them are synced from the primary. This commit does two things: 1) Updates inactive_since for sync slots with the value received from the primary's slot. 2) Ensures the value is set to current timestamp during the shutdown of slot sync machinery to help correctly interpret the time if the standby gets promoted without a restart. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACWLctoiH-pSjWnEpR54q4DED6rw_BRJm5pCx86_Y01MoQ%40mail.gmail.com --- doc/src/sgml/system-views.sgml | 9 +++ src/backend/replication/logical/slotsync.c | 61 ++++++++++++++++++- src/backend/replication/slot.c | 40 ++++++++---- src/test/perl/PostgreSQL/Test/Cluster.pm | 34 +++++++++++ src/test/recovery/t/019_replslot_limit.pl | 26 +------- .../t/040_standby_failover_slots_sync.pl | 43 +++++++++++++ 6 files changed, 173 insertions(+), 40 deletions(-) diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 3c8dca8ca3..c8d97ab375 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2530,6 +2530,15 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <para> The time since the slot has become inactive. <literal>NULL</literal> if the slot is currently being used. + Note that the slots on the standbys that are being synced from a + primary server (whose <structfield>synced</structfield> field is + <literal>true</literal>), will get the + <structfield>inactive_since</structfield> value from the + corresponding remote slot on the primary. Also, after the standby + starts up, the <structfield>inactive_since</structfield> value + (for such synced slots) will remain <literal>NULL</literal> until + the next synchronization (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>). </para></entry> </row> diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 30480960c5..9c95a4b062 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -137,9 +137,12 @@ typedef struct RemoteSlot /* RS_INVAL_NONE if valid, or the reason of invalidation */ ReplicationSlotInvalidationCause invalidated; + + TimestampTz inactive_since; /* in seconds */ } RemoteSlot; static void slotsync_failure_callback(int code, Datum arg); +static void update_synced_slots_inactive_since(void); /* * If necessary, update the local synced slot's metadata based on the data @@ -167,6 +170,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) remote_slot->two_phase == slot->data.two_phase && remote_slot->failover == slot->data.failover && remote_slot->confirmed_lsn == slot->data.confirmed_flush && + remote_slot->inactive_since == slot->inactive_since && strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) return false; @@ -182,6 +186,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) slot->data.confirmed_flush = remote_slot->confirmed_lsn; slot->data.catalog_xmin = remote_slot->catalog_xmin; slot->effective_catalog_xmin = remote_slot->catalog_xmin; + slot->inactive_since = remote_slot->inactive_since; SpinLockRelease(&slot->mutex); if (xmin_changed) @@ -652,9 +657,9 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) static bool synchronize_slots(WalReceiverConn *wrconn) { -#define SLOTSYNC_COLUMN_COUNT 9 +#define SLOTSYNC_COLUMN_COUNT 10 Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, - LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID, TIMESTAMPTZOID}; WalRcvExecResult *res; TupleTableSlot *tupslot; @@ -663,7 +668,7 @@ synchronize_slots(WalReceiverConn *wrconn) bool started_tx = false; const char *query = "SELECT slot_name, plugin, confirmed_flush_lsn," " restart_lsn, catalog_xmin, two_phase, failover," - " database, invalidation_reason" + " database, invalidation_reason, inactive_since" " FROM pg_catalog.pg_replication_slots" " WHERE failover and NOT temporary"; @@ -743,6 +748,13 @@ synchronize_slots(WalReceiverConn *wrconn) remote_slot->invalidated = isnull ? RS_INVAL_NONE : GetSlotInvalidationCause(TextDatumGetCString(d)); + /* + * It is possible to get null value for inactive_since if the slot is + * active on the primary server, so handle accordingly. + */ + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->inactive_since = isnull ? 0 : DatumGetTimestampTz(d); + /* Sanity check */ Assert(col == SLOTSYNC_COLUMN_COUNT); @@ -1296,6 +1308,46 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len) Assert(false); } +/* + * Update the inactive_since property for synced slots. + */ +static void +update_synced_slots_inactive_since(void) +{ + TimestampTz now = 0; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + + /* + * We get the current time beforehand and only once to avoid + * system calls overhead while holding the lock. + */ + if (now == 0) + now = GetCurrentTimestamp(); + + /* + * Set the time since the slot has become inactive after shutting + * down slot sync machinery. This helps correctly interpret the + * time if the standby gets promoted without a restart. + */ + SpinLockAcquire(&s->mutex); + s->inactive_since = now; + SpinLockRelease(&s->mutex); + } + } + + LWLockRelease(ReplicationSlotControlLock); +} + /* * Shut down the slot sync worker. */ @@ -1309,6 +1361,7 @@ ShutDownSlotSync(void) if (SlotSyncCtx->pid == InvalidPid) { SpinLockRelease(&SlotSyncCtx->mutex); + update_synced_slots_inactive_since(); return; } SpinLockRelease(&SlotSyncCtx->mutex); @@ -1341,6 +1394,8 @@ ShutDownSlotSync(void) } SpinLockRelease(&SlotSyncCtx->mutex); + + update_synced_slots_inactive_since(); } /* diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index d778c0b921..7dbb44b7b0 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -42,6 +42,7 @@ #include "access/transam.h" #include "access/xlog_internal.h" #include "access/xlogrecovery.h" +#include "access/xlogutils.h" #include "common/file_utils.h" #include "common/string.h" #include "miscadmin.h" @@ -655,6 +656,7 @@ ReplicationSlotRelease(void) char *slotname = NULL; /* keep compiler quiet */ bool is_logical = false; /* keep compiler quiet */ TimestampTz now = 0; + bool is_slot_being_synced = false; Assert(slot != NULL && slot->active_pid != 0); @@ -690,12 +692,15 @@ ReplicationSlotRelease(void) } /* - * Set the last inactive time after marking the slot inactive. We don't - * set it for the slots currently being synced from the primary to the - * standby because such slots are typically inactive as decoding is not - * allowed on those. + * Set the time since the slot has become inactive. + * + * Note that we don't set it for the slots currently being synced from the + * primary to the standby, because such slots typically sync the data from + * the remote slot. */ - if (!(RecoveryInProgress() && slot->data.synced)) + if (RecoveryInProgress() && slot->data.synced) + is_slot_being_synced = true; + else now = GetCurrentTimestamp(); if (slot->data.persistency == RS_PERSISTENT) @@ -706,11 +711,12 @@ ReplicationSlotRelease(void) */ SpinLockAcquire(&slot->mutex); slot->active_pid = 0; - slot->inactive_since = now; + if (!is_slot_being_synced) + slot->inactive_since = now; SpinLockRelease(&slot->mutex); ConditionVariableBroadcast(&slot->active_cv); } - else + else if (!is_slot_being_synced) { SpinLockAcquire(&slot->mutex); slot->inactive_since = now; @@ -2369,13 +2375,21 @@ RestoreSlotFromDisk(const char *name) slot->active_pid = 0; /* - * We set the last inactive time after loading the slot from the disk - * into memory. Whoever acquires the slot i.e. makes the slot active - * will reset it. We don't set it for the slots currently being synced - * from the primary to the standby because such slots are typically - * inactive as decoding is not allowed on those. + * Set the time since the slot has become inactive after loading the + * slot from the disk into memory. Whoever acquires the slot i.e. + * makes the slot active will reset it. + * + * Note that we don't set it for the slots currently being synced from + * the primary to the standby, because such slots typically sync the + * data from the remote slot. We use InRecovery flag instead of + * RecoveryInProgress() as the latter always returns true at this time + * even on primary. + * + * Note that for synced slots after the standby starts up (i.e. after + * the slots are loaded from the disk), the inactive_since will remain + * zero until the next slot sync cycle. */ - if (!(RecoveryInProgress() && slot->data.synced)) + if (!(InRecovery && slot->data.synced)) slot->inactive_since = GetCurrentTimestamp(); else slot->inactive_since = 0; diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index b08296605c..ddfc3236f3 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -3276,6 +3276,40 @@ sub create_logical_slot_on_standby =pod +=item $node->get_slot_inactive_since_value(self, slot_name, reference_time) + +Get inactive_since column value for a given replication slot validating it +against optional reference time. + +=cut + +sub get_slot_inactive_since_value +{ + my ($self, $slot_name, $reference_time) = @_; + my $name = $self->name; + + my $inactive_since = $self->safe_psql('postgres', + qq(SELECT inactive_since FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) + ); + + # Check that the captured time is sane + if (defined $reference_time) + { + is($self->safe_psql('postgres', + qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND + '$inactive_since'::timestamptz >= '$reference_time'::timestamptz;] + ), + 't', + "last inactive time for slot $slot_name is valid on node $name") + or die "could not validate captured inactive_since for slot $slot_name"; + } + + return $inactive_since; +} + +=pod + =item $node->advance_wal(num) Advance WAL of node by given number of segments. diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl index 3b9a306a8b..c8e5e5054d 100644 --- a/src/test/recovery/t/019_replslot_limit.pl +++ b/src/test/recovery/t/019_replslot_limit.pl @@ -443,7 +443,7 @@ $primary4->safe_psql( # Get inactive_since value after the slot's creation. Note that the slot is # still inactive till it's used by the standby below. my $inactive_since = - capture_and_validate_slot_inactive_since($primary4, $sb4_slot, $slot_creation_time); + $primary4->get_slot_inactive_since_value($sb4_slot, $slot_creation_time); $standby4->start; @@ -502,7 +502,7 @@ $publisher4->safe_psql('postgres', # Get inactive_since value after the slot's creation. Note that the slot is # still inactive till it's used by the subscriber below. $inactive_since = - capture_and_validate_slot_inactive_since($publisher4, $lsub4_slot, $slot_creation_time); + $publisher4->get_slot_inactive_since_value($lsub4_slot, $slot_creation_time); $subscriber4->start; $subscriber4->safe_psql('postgres', @@ -540,26 +540,4 @@ is( $publisher4->safe_psql( $publisher4->stop; $subscriber4->stop; -# Capture and validate inactive_since of a given slot. -sub capture_and_validate_slot_inactive_since -{ - my ($node, $slot_name, $slot_creation_time) = @_; - - my $inactive_since = $node->safe_psql('postgres', - qq(SELECT inactive_since FROM pg_replication_slots - WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) - ); - - # Check that the captured time is sane - is( $node->safe_psql( - 'postgres', - qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND - '$inactive_since'::timestamptz >= '$slot_creation_time'::timestamptz;] - ), - 't', - "last inactive time for an active slot $slot_name is sane"); - - return $inactive_since; -} - done_testing(); diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index f47bfd78eb..33e3a8dcf0 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1'); $subscriber1->init; $subscriber1->start; +# Capture the time before the logical failover slot is created on the +# primary. We later call this publisher as primary anyway. +my $slot_creation_time_on_primary = $publisher->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + # Create a slot on the publisher with failover disabled $publisher->safe_psql('postgres', "SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);" @@ -174,6 +181,10 @@ $primary->poll_query_until( "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", 1); +# Capture the inactive_since of the slot from the primary +my $inactive_since_on_primary = + $primary->get_slot_inactive_since_value('lsub1_slot', $slot_creation_time_on_primary); + # Wait for the standby to catch up so that the standby is not lagging behind # the subscriber. $primary->wait_for_replay_catchup($standby1); @@ -190,6 +201,18 @@ is( $standby1->safe_psql( "t", 'logical slots have synced as true on standby'); +# Capture the inactive_since of the synced slot on the standby +my $inactive_since_on_standby = + $standby1->get_slot_inactive_since_value('lsub1_slot', $slot_creation_time_on_primary); + +# Synced slots on the standby must get the inactive_since from the primary. +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_primary'::timestamptz = '$inactive_since_on_standby'::timestamptz;" + ), + "t", + 'synchronized slot has got the inactive_since from the primary'); + ################################################## # Test that the synchronized slot will be dropped if the corresponding remote # slot on the primary server has been dropped. @@ -750,8 +773,28 @@ $primary->reload; $standby1->start; $primary->wait_for_replay_catchup($standby1); +# Capture the time before the standby is promoted +my $promotion_time_on_primary = $standby1->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + $standby1->promote; +# Capture the inactive_since of the synced slot after the promotion. +# Expectation here is that the slot gets its own inactive_since as part of the +# promotion. We do this check before the slot is enabled on the new primary +# below, otherwise the slot gets active setting inactive_since to NULL. +my $inactive_since_on_new_primary = + $standby1->get_slot_inactive_since_value('lsub1_slot', $promotion_time_on_primary); + +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_new_primary'::timestamptz > '$inactive_since_on_primary'::timestamptz" + ), + "t", + 'synchronized slot has got its own inactive_since on the new primary'); + # Update subscription with the new primary's connection info my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; $subscriber1->safe_psql('postgres', -- 2.34.1 [application/octet-stream] v29-0002-Add-inactive_timeout-based-replication-slot-inva.patch (29.4K, ../../CALj2ACX1bdaX2gf5ttwk3buiDrgZVV=vM6S1SQKf=Sxhj1HkzQ@mail.gmail.com/3-v29-0002-Add-inactive_timeout-based-replication-slot-inva.patch) download | inline diff: From 88ab4ed8346103b61c37d471db40ea04f2837596 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Wed, 27 Mar 2024 15:28:40 +0000 Subject: [PATCH v29 2/2] Add inactive_timeout based replication slot invalidation. Till now, postgres has the ability to invalidate inactive replication slots based on the amount of WAL (set via max_slot_wal_keep_size GUC) that will be needed for the slots in case they become active. However, choosing a default value for max_slot_wal_keep_size is tricky. Because the amount of WAL a customer generates, and their allocated storage will vary greatly in production, making it difficult to pin down a one-size-fits-all value. It is often easy for developers to set a timeout of say 1 or 2 or 3 days at slot level, after which the inactive slots get dropped. To achieve the above, postgres introduces a GUC allowing users set inactive timeout and then a slot stays inactive for this much amount of time it invalidates the slot. The invalidation check happens at various locations to help being as latest as possible, these locations include the following: - Whenever the slot is acquired and the slot acquisition errors out if invalidated. - During checkpoint Note that this new invalidation mechanism won't kick-in for the slots that are currently being synced from the primary to the standby. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com --- doc/src/sgml/config.sgml | 18 ++ doc/src/sgml/system-views.sgml | 7 + .../replication/logical/logicalfuncs.c | 2 +- src/backend/replication/logical/slotsync.c | 4 +- src/backend/replication/slot.c | 171 ++++++++++- src/backend/replication/slotfuncs.c | 2 +- src/backend/replication/walsender.c | 4 +- src/backend/utils/adt/pg_upgrade_support.c | 2 +- src/backend/utils/misc/guc_tables.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/replication/slot.h | 9 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/050_invalidate_slots.pl | 278 ++++++++++++++++++ 13 files changed, 499 insertions(+), 12 deletions(-) create mode 100644 src/test/recovery/t/050_invalidate_slots.pl diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 65a6e6c408..db240c5b44 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4544,6 +4544,24 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows </listitem> </varlistentry> + <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout"> + <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>) + <indexterm> + <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Invalidate replication slots that are inactive for longer than this + amount of time. If this value is specified without units, it is taken + as seconds. A value of zero (which is default) disables the timeout + mechanism. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server + command line. + </para> + </listitem> + </varlistentry> + <varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp"> <term><varname>track_commit_timestamp</varname> (<type>boolean</type>) <indexterm> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index c8d97ab375..5c05fd1c07 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2582,6 +2582,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx perform logical decoding. It is set only for logical slots. </para> </listitem> + <listitem> + <para> + <literal>inactive_timeout</literal> means that the slot has been + inactive for the duration specified by + <xref linkend="guc-replication-slot-inactive-timeout"/> parameter. + </para> + </listitem> </itemizedlist> </para></entry> </row> diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c index b4dd5cce75..56fc1a45a9 100644 --- a/src/backend/replication/logical/logicalfuncs.c +++ b/src/backend/replication/logical/logicalfuncs.c @@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin else end_of_wal = GetXLogReplayRecPtr(NULL); - ReplicationSlotAcquire(NameStr(*name), true); + ReplicationSlotAcquire(NameStr(*name), true, true); PG_TRY(); { diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 9c95a4b062..71b6a254cf 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -321,7 +321,7 @@ drop_local_obsolete_slots(List *remote_slot_list) if (synced_slot) { - ReplicationSlotAcquire(NameStr(local_slot->data.name), true); + ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false); ReplicationSlotDropAcquired(); } @@ -531,7 +531,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) * InvalidatePossiblyObsoleteSlot() where it invalidates slot directly * if the slot is not acquired by other processes. */ - ReplicationSlotAcquire(remote_slot->name, true); + ReplicationSlotAcquire(remote_slot->name, true, false); Assert(slot == MyReplicationSlot); diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 7dbb44b7b0..0c5a601d87 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -108,10 +108,11 @@ const char *const SlotInvalidationCauses[] = { [RS_INVAL_WAL_REMOVED] = "wal_removed", [RS_INVAL_HORIZON] = "rows_removed", [RS_INVAL_WAL_LEVEL] = "wal_level_insufficient", + [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout", }; /* Maximum number of invalidation causes */ -#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL +#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1), "array length mismatch"); @@ -141,6 +142,7 @@ ReplicationSlot *MyReplicationSlot = NULL; /* GUC variables */ int max_replication_slots = 10; /* the maximum number of replication * slots */ +int replication_slot_inactive_timeout = 0; /* * This GUC lists streaming replication standby server slot names that @@ -159,6 +161,8 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr; static void ReplicationSlotShmemExit(int code, Datum arg); static void ReplicationSlotDropPtr(ReplicationSlot *slot); +static bool InvalidateSlotForInactiveTimeout(ReplicationSlot *slot, + bool need_locks); /* internal persistency functions */ static void RestoreSlotFromDisk(const char *name); @@ -536,9 +540,13 @@ ReplicationSlotName(int index, Name name) * * An error is raised if nowait is true and the slot is currently in use. If * nowait is false, we sleep until the slot is released by the owning process. + * + * If check_for_invalidation is true, the slot is checked for invalidation + * based on replication_slot_inactive_timeout GUC and an error is raised after making the slot ours. */ void -ReplicationSlotAcquire(const char *name, bool nowait) +ReplicationSlotAcquire(const char *name, bool nowait, + bool check_for_invalidation) { ReplicationSlot *s; int active_pid; @@ -616,6 +624,41 @@ retry: /* We made this slot active, so it's ours now. */ MyReplicationSlot = s; + /* + * Check if the given slot can be invalidated based on its inactive + * timeout. If yes, persist the invalidated state to disk and then error + * out. We do this only after making the slot ours to avoid anyone else + * acquiring it while we check for its invalidation. + */ + if (check_for_invalidation) + { + /* The slot is ours by now */ + Assert(s->active_pid == MyProcPid); + + /* + * Well, the slot is not yet ours really unless we check for the + * invalidation below. + */ + s->active_pid = 0; + if (InvalidateReplicationSlotForInactiveTimeout(s, true, true)) + { + /* + * If the slot has been invalidated, recalculate the resource + * limits. + */ + ReplicationSlotsComputeRequiredXmin(false); + ReplicationSlotsComputeRequiredLSN(); + + /* Might need it for slot clean up on error, so restore it */ + s->active_pid = MyProcPid; + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot acquire invalidated replication slot \"%s\"", + NameStr(MyReplicationSlot->data.name)))); + } + s->active_pid = MyProcPid; + } + /* * The call to pgstat_acquire_replslot() protects against stats for a * different slot, from before a restart or such, being present during @@ -790,7 +833,7 @@ ReplicationSlotDrop(const char *name, bool nowait) { Assert(MyReplicationSlot == NULL); - ReplicationSlotAcquire(name, nowait); + ReplicationSlotAcquire(name, nowait, false); /* * Do not allow users to drop the slots which are currently being synced @@ -813,7 +856,7 @@ ReplicationSlotAlter(const char *name, bool failover) { Assert(MyReplicationSlot == NULL); - ReplicationSlotAcquire(name, false); + ReplicationSlotAcquire(name, false, true); if (SlotIsPhysical(MyReplicationSlot)) ereport(ERROR, @@ -1515,6 +1558,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause, case RS_INVAL_WAL_LEVEL: appendStringInfoString(&err_detail, _("Logical decoding on standby requires wal_level >= logical on the primary server.")); break; + case RS_INVAL_INACTIVE_TIMEOUT: + appendStringInfoString(&err_detail, _("The slot has been inactive for more than the time specified by replication_slot_inactive_timeout parameter.")); + break; case RS_INVAL_NONE: pg_unreachable(); } @@ -1628,6 +1674,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, if (SlotIsLogical(s)) invalidation_cause = cause; break; + case RS_INVAL_INACTIVE_TIMEOUT: + if (InvalidateReplicationSlotForInactiveTimeout(s, false, false)) + invalidation_cause = cause; + break; case RS_INVAL_NONE: pg_unreachable(); } @@ -1781,6 +1831,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, * - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given * db; dboid may be InvalidOid for shared relations * - RS_INVAL_WAL_LEVEL: is logical + * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout occurs * * NB - this runs as part of checkpoint, so avoid raising errors if possible. */ @@ -1832,6 +1883,103 @@ restart: return invalidated; } +/* + * Invalidate given slot based on replication_slot_inactive_timeout GUC. + * + * Returns true if the slot has got invalidated. + * + * NB - this function also runs as part of checkpoint, so avoid raising errors + * if possible. + */ +bool +InvalidateReplicationSlotForInactiveTimeout(ReplicationSlot *slot, + bool need_locks, + bool persist_state) +{ + if (!InvalidateSlotForInactiveTimeout(slot, need_locks)) + return false; + + Assert(slot->active_pid == 0); + + SpinLockAcquire(&slot->mutex); + slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT; + + /* Make sure the invalidated state persists across server restart */ + slot->just_dirtied = true; + slot->dirty = true; + SpinLockRelease(&slot->mutex); + + if (persist_state) + { + char path[MAXPGPATH]; + + sprintf(path, "pg_replslot/%s", NameStr(slot->data.name)); + SaveSlotToPath(slot, path, ERROR); + } + + ReportSlotInvalidation(RS_INVAL_INACTIVE_TIMEOUT, false, 0, + slot->data.name, InvalidXLogRecPtr, + InvalidXLogRecPtr, InvalidTransactionId); + + return true; +} + +/* + * Helper for InvalidateReplicationSlotForInactiveTimeout + */ +static bool +InvalidateSlotForInactiveTimeout(ReplicationSlot *slot, bool need_locks) +{ + ReplicationSlotInvalidationCause inavidation_cause = RS_INVAL_NONE; + + if (slot->inactive_since == 0 || + replication_slot_inactive_timeout == 0) + return false; + + /* + * Do not invalidate the slots which are currently being synced from the + * primary to the standby. + */ + if (RecoveryInProgress() && slot->data.synced) + return false; + + if (need_locks) + { + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + SpinLockAcquire(&slot->mutex); + } + + Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED)); + + /* + * Check if the slot needs to be invalidated due to + * replication_slot_inactive_timeout GUC. We do this with the spinlock + * held to avoid race conditions -- for example the restart_lsn could move + * forward, or the slot could be dropped. + */ + if (slot->inactive_since > 0 && + replication_slot_inactive_timeout > 0) + { + TimestampTz now; + + /* inactive_since is only tracked for inactive slots */ + Assert(slot->active_pid == 0); + + now = GetCurrentTimestamp(); + if (TimestampDifferenceExceeds(slot->inactive_since, now, + replication_slot_inactive_timeout * 1000)) + inavidation_cause = RS_INVAL_INACTIVE_TIMEOUT; + } + + if (need_locks) + { + SpinLockRelease(&slot->mutex); + LWLockRelease(ReplicationSlotControlLock); + } + + return (inavidation_cause == RS_INVAL_INACTIVE_TIMEOUT); +} + /* * Flush all replication slots to disk. * @@ -1844,6 +1992,7 @@ void CheckPointReplicationSlots(bool is_shutdown) { int i; + bool invalidated = false; elog(DEBUG1, "performing replication slot checkpoint"); @@ -1867,6 +2016,13 @@ CheckPointReplicationSlots(bool is_shutdown) /* save the slot to disk, locking is handled in SaveSlotToPath() */ sprintf(path, "pg_replslot/%s", NameStr(s->data.name)); + /* + * Here's an opportunity to invalidate inactive replication slots + * based on timeout, so let's do it. + */ + if (InvalidateReplicationSlotForInactiveTimeout(s, true, false)) + invalidated = true; + /* * Slot's data is not flushed each time the confirmed_flush LSN is * updated as that could lead to frequent writes. However, we decide @@ -1893,6 +2049,13 @@ CheckPointReplicationSlots(bool is_shutdown) SaveSlotToPath(s, path, LOG); } LWLockRelease(ReplicationSlotAllocationLock); + + /* If the slot has been invalidated, recalculate the resource limits */ + if (invalidated) + { + ReplicationSlotsComputeRequiredXmin(false); + ReplicationSlotsComputeRequiredLSN(); + } } /* diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index da57177c25..677c0bf0a2 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -651,7 +651,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS) moveto = Min(moveto, GetXLogReplayRecPtr(NULL)); /* Acquire the slot so we "own" it */ - ReplicationSlotAcquire(NameStr(*slotname), true); + ReplicationSlotAcquire(NameStr(*slotname), true, true); /* A slot whose restart_lsn has never been reserved cannot be advanced */ if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn)) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index bc40c454de..96eeb8b7d2 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd) if (cmd->slotname) { - ReplicationSlotAcquire(cmd->slotname, true); + ReplicationSlotAcquire(cmd->slotname, true, true); if (SlotIsLogical(MyReplicationSlot)) ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), @@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) Assert(!MyReplicationSlot); - ReplicationSlotAcquire(cmd->slotname, true); + ReplicationSlotAcquire(cmd->slotname, true, true); /* * Force a disconnect, so that the decoding code doesn't need to care diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index c54b08fe18..82956d58d3 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS) slot_name = PG_GETARG_NAME(0); /* Acquire the given slot */ - ReplicationSlotAcquire(NameStr(*slot_name), true); + ReplicationSlotAcquire(NameStr(*slot_name), true, false); Assert(SlotIsLogical(MyReplicationSlot)); diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 1e71e7db4a..c97a2d83a7 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -2954,6 +2954,18 @@ struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING, + gettext_noop("Sets the amount of time to wait before invalidating an " + "inactive replication slot."), + NULL, + GUC_UNIT_S + }, + &replication_slot_inactive_timeout, + 0, 0, INT_MAX, + NULL, NULL, NULL + }, + { {"commit_delay", PGC_SUSET, WAL_SETTINGS, gettext_noop("Sets the delay in microseconds between transaction commit and " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 2244ee52f7..622cdf503d 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -334,6 +334,7 @@ #wal_sender_timeout = 60s # in milliseconds; 0 disables #track_commit_timestamp = off # collect timestamp of transaction commit # (change requires restart) +#replication_slot_inactive_timeout = 0 # in seconds; 0 disables # - Primary Server - diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index 7b937d1a0c..cd98ab5112 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause RS_INVAL_HORIZON, /* wal_level insufficient for slot */ RS_INVAL_WAL_LEVEL, + /* inactive slot timeout has occurred */ + RS_INVAL_INACTIVE_TIMEOUT, } ReplicationSlotInvalidationCause; extern PGDLLIMPORT const char *const SlotInvalidationCauses[]; @@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot; /* GUCs */ extern PGDLLIMPORT int max_replication_slots; extern PGDLLIMPORT char *standby_slot_names; +extern PGDLLIMPORT int replication_slot_inactive_timeout; /* shmem initialization functions */ extern Size ReplicationSlotsShmemSize(void); @@ -245,7 +248,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait); extern void ReplicationSlotDropAcquired(void); extern void ReplicationSlotAlter(const char *name, bool failover); -extern void ReplicationSlotAcquire(const char *name, bool nowait); +extern void ReplicationSlotAcquire(const char *name, bool nowait, + bool check_for_invalidation); extern void ReplicationSlotRelease(void); extern void ReplicationSlotCleanup(void); extern void ReplicationSlotSave(void); @@ -264,6 +268,9 @@ extern bool InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause XLogSegNo oldestSegno, Oid dboid, TransactionId snapshotConflictHorizon); +extern bool InvalidateReplicationSlotForInactiveTimeout(ReplicationSlot *slot, + bool need_locks, + bool persist_state); extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_lock); extern int ReplicationSlotIndex(ReplicationSlot *slot); extern bool ReplicationSlotName(int index, Name name); diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index b1eb77b1ec..708a2a3798 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -51,6 +51,7 @@ tests += { 't/040_standby_failover_slots_sync.pl', 't/041_checkpoint_at_promote.pl', 't/042_low_level_backup.pl', + 't/050_invalidate_slots.pl', ], }, } diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl new file mode 100644 index 0000000000..cf233c5513 --- /dev/null +++ b/src/test/recovery/t/050_invalidate_slots.pl @@ -0,0 +1,278 @@ +# Copyright (c) 2024, PostgreSQL Global Development Group + +# Test for replication slots invalidation +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Utils; +use PostgreSQL::Test::Cluster; +use Test::More; +use Time::HiRes qw(usleep); + +# ============================================================================= +# Testcase start: Invalidate streaming standby's slot as well as logical +# failover slot on primary due to inactive timeout GUC. Also, check the logical +# failover slot synced on to the standby doesn't invalidate the slot on its own, +# but gets the invalidated state from the remote slot on the primary. + +# Initialize primary node +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init(allows_streaming => 'logical'); + +# Avoid checkpoint during the test, otherwise, the test can get unpredictable +$primary->append_conf( + 'postgresql.conf', q{ +checkpoint_timeout = 1h +autovacuum = off +}); +$primary->start; + +# Take backup +my $backup_name = 'my_backup'; +$primary->backup($backup_name); + +# Create a standby linking to the primary using the replication slot +my $standby1 = PostgreSQL::Test::Cluster->new('standby1'); +$standby1->init_from_backup($primary, $backup_name, has_streaming => 1); + +my $connstr_1 = $primary->connstr; +$standby1->append_conf( + 'postgresql.conf', qq( +hot_standby_feedback = on +primary_slot_name = 'sb1_slot' +primary_conninfo = '$connstr_1 dbname=postgres' +)); + +# Create sync slot on the primary +$primary->psql('postgres', + q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);} +); + +$primary->safe_psql( + 'postgres', qq[ + SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot'); +]); + +$standby1->start; + +my $standby1_logstart = -s $standby1->logfile; + +# Wait until standby has replayed enough data +$primary->wait_for_catchup($standby1); + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the logical failover slot is created on the standby and is +# flagged as 'synced'. +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 1 FROM pg_replication_slots + WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;} + ), + "t", + 'logical slot has synced as true on standby'); + +my $logstart = -s $primary->logfile; + +# Set timeout so that the next checkpoint will invalidate the inactive +# replication slot. +$primary->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s'; +]); +$primary->reload; + +# Wait for the logical failover slot to become inactive on the primary. Note +# that nobody has acquired that slot yet, so due to inactive timeout setting +# above it must get invalidated. +wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart); + +# Set timeout on the standby also to check the synced slots don't get +# invalidated due to timeout on the standby. +$standby1->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s'; +]); +$standby1->reload; + +# Now, sync the logical failover slot from the remote slot on the primary. +# Note that the remote slot has already been invalidated due to inactive +# timeout. Now, the standby must also see it as invalidated. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Wait for the inactive replication slot to be invalidated. +$standby1->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE slot_name = 'lsub1_sync_slot' AND + invalidation_reason = 'inactive_timeout'; +]) + or die + "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby"; + +# Synced slot mustn't get invalidated on the standby, it must sync invalidation +# from the primary. So, we must not see the slot's invalidation message in server +# log. +ok( !$standby1->log_contains( + "invalidating obsolete replication slot \"lsub1_sync_slot\"", + $standby1_logstart), + 'check that syned slot has not been invalidated on the standby'); + +# Stop standby to make the standby's replication slot on the primary inactive +$standby1->stop; + +# Wait for the standby's replication slot to become inactive +wait_for_slot_invalidation($primary, 'sb1_slot', $logstart); + +# Testcase end: Invalidate streaming standby's slot as well as logical failover +# slot on primary due to inactive timeout GUC. Also, check the logical failover +# slot synced on to the standby doesn't invalidate the slot on its own, but +# gets the invalidated state from the remote slot on the primary. +# ============================================================================= + +# ============================================================================= +# Testcase start: Invalidate logical subscriber's slot due to inactive timeout +# GUC. + +my $publisher = $primary; + +# Prepare for the next test +$publisher->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '0'; +]); +$publisher->reload; + +# Create subscriber node +my $subscriber = PostgreSQL::Test::Cluster->new('sub'); +$subscriber->init; +$subscriber->start; + +# Create tables +$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)"); +$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)"); + +# Insert some data +$subscriber->safe_psql('postgres', + "INSERT INTO test_tbl VALUES (generate_series(1, 5));"); + +# Setup logical replication +my $publisher_connstr = $publisher->connstr . ' dbname=postgres'; +$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES"); +$publisher->safe_psql( + 'postgres', qq[ + SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput'); +]); + +$subscriber->safe_psql('postgres', + "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)" +); + +$subscriber->wait_for_subscription_sync($publisher, 'sub'); + +my $result = + $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl"); + +is($result, qq(5), "check initial copy was done"); + +# Prepare for the next test +$publisher->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s'; +]); +$publisher->reload; + +$logstart = -s $publisher->logfile; + +# Stop subscriber to make the replication slot on publisher inactive +$subscriber->stop; + +# Wait for the replication slot to become inactive and then invalidated due to +# timeout. +wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart); + +# Testcase end: Invalidate logical subscriber's slot due to inactive timeout +# GUC. +# ============================================================================= + +# ============================================================================= +# Start: Helper functions used for this test file + +sub wait_for_slot_invalidation +{ + my ($node, $slot_name, $offset) = @_; + my $name = $node->name; + + # Wait for the replication slot to become inactive + $node->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND active = 'f'; + ]) + or die + "Timed out while waiting for replication slot to become inactive"; + + # Wait for the replication slot info to be updated + $node->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE inactive_since IS NOT NULL + AND slot_name = '$slot_name' AND active = 'f'; + ]) + or die + "Timed out while waiting for info of replication slot $slot_name to be updated on node $name"; + + check_for_slot_invalidation_in_server_log($node, $slot_name, $offset); + + # Wait for the inactive replication slot to be invalidated. + $node->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND + invalidation_reason = 'inactive_timeout'; + ]) + or die + "Timed out while waiting for inactive replication slot $slot_name to be invalidated on node $name"; + + # Check that the invalidated slot cannot be acquired + my ($result, $stdout, $stderr); + + ($result, $stdout, $stderr) = $primary->psql( + 'postgres', qq[ + SELECT pg_replication_slot_advance('$slot_name', '0/1'); + ]); + + ok( $stderr =~ /cannot acquire invalidated replication slot "$slot_name"/, + "detected error upon trying to acquire invalidated slot $slot_name on node $name" + ) + or die + "could not detect error upon trying to acquire invalidated slot $slot_name"; +} + +# Check for invalidation of slot in server log. +sub check_for_slot_invalidation_in_server_log +{ + my ($node, $slot_name, $offset) = @_; + my $invalidated = 0; + + for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++) + { + $node->safe_psql('postgres', "CHECKPOINT"); + if ($node->log_contains( + "invalidating obsolete replication slot \"$slot_name\"", + $offset)) + { + $invalidated = 1; + last; + } + usleep(100_000); + } + ok($invalidated, + "check that slot $slot_name invalidation has been logged"); +} + +# ============================================================================= +# End: Helper functions used for this test file + +done_testing(); -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-27 16:12 Bertrand Drouvot <[email protected]> parent: Bharath Rupireddy <[email protected]> 3 siblings, 0 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-27 16:12 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Wed, Mar 27, 2024 at 09:00:37PM +0530, Bharath Rupireddy wrote: > On Wed, Mar 27, 2024 at 6:54 PM Bertrand Drouvot > <[email protected]> wrote: > > > > Hi, > > > > On Wed, Mar 27, 2024 at 05:55:05PM +0530, Bharath Rupireddy wrote: > > > On Wed, Mar 27, 2024 at 3:42 PM Bertrand Drouvot > > > Please see the attached v28 patch. > > > > Thanks! > > > > 1 === sorry I missed it in the previous review > > > > if (!(RecoveryInProgress() && slot->data.synced)) > > + { > > now = GetCurrentTimestamp(); > > + update_inactive_since = true; > > + } > > + else > > + update_inactive_since = false; > > > > I think update_inactive_since is not needed, we could rely on (now > 0) instead. > > Thought of using it, but, at the expense of readability. I prefer to > use a variable instead. That's fine too. > However, I changed the variable to be more meaningful to is_slot_being_synced. Yeah makes sense and even easier to read. v29-0001 LGTM. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-28 03:46 shveta malik <[email protected]> parent: Bharath Rupireddy <[email protected]> 3 siblings, 0 replies; 46+ messages in thread From: shveta malik @ 2024-03-28 03:46 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Wed, Mar 27, 2024 at 9:00 PM Bharath Rupireddy <[email protected]> wrote: > > Thanks. I'm attaching v29 patches. 0001 managing inactive_since on the > standby for sync slots. 0002 implementing inactive timeout GUC based > invalidation mechanism. > > Please have a look. Thanks for the patches. v29-001 looks good to me. thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-28 09:43 Bertrand Drouvot <[email protected]> parent: Bharath Rupireddy <[email protected]> 3 siblings, 1 reply; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-28 09:43 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Wed, Mar 27, 2024 at 09:00:37PM +0530, Bharath Rupireddy wrote: > standby for sync slots. 0002 implementing inactive timeout GUC based > invalidation mechanism. > > Please have a look. Thanks! Regarding 0002: Some testing: T1 === When the slot is invalidated on the primary, then the reason is propagated to the sync slot (if any). That's fine but we are loosing the inactive_since on the standby: Primary: postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot'; slot_name | inactive_since | conflicting | invalidation_reason -------------+-------------------------------+-------------+--------------------- lsub29_slot | 2024-03-28 08:24:51.672528+00 | f | inactive_timeout (1 row) Standby: postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot'; slot_name | inactive_since | conflicting | invalidation_reason -------------+----------------+-------------+--------------------- lsub29_slot | | f | inactive_timeout (1 row) I think in this case it should always reflect the value from the primary (so that one can understand why it is invalidated). T2 === And it is set to a value during promotion: postgres=# select pg_promote(); pg_promote ------------ t (1 row) postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot'; slot_name | inactive_since | conflicting | invalidation_reason -------------+------------------------------+-------------+--------------------- lsub29_slot | 2024-03-28 08:30:11.74505+00 | f | inactive_timeout (1 row) I think when it is invalidated it should always reflect the value from the primary (so that one can understand why it is invalidated). T3 === As far the slot invalidation on the primary: postgres=# SELECT * FROM pg_logical_slot_get_changes('lsub29_slot', NULL, NULL, 'include-xids', '0'); ERROR: cannot acquire invalidated replication slot "lsub29_slot" Can we make the message more consistent with what can be found in CreateDecodingContext() for example? T4 === Also, it looks like querying pg_replication_slots() does not trigger an invalidation: I think it should if the slot is not invalidated yet (and matches the invalidation criteria). Code review: CR1 === + Invalidate replication slots that are inactive for longer than this + amount of time. If this value is specified without units, it is taken s/Invalidate/Invalidates/? Should we mention the relationship with inactive_since? CR2 === + * + * If check_for_invalidation is true, the slot is checked for invalidation + * based on replication_slot_inactive_timeout GUC and an error is raised after making the slot ours. */ void -ReplicationSlotAcquire(const char *name, bool nowait) +ReplicationSlotAcquire(const char *name, bool nowait, + bool check_for_invalidation) s/check_for_invalidation/check_for_timeout_invalidation/? CR3 === + if (slot->inactive_since == 0 || + replication_slot_inactive_timeout == 0) + return false; Better to test replication_slot_inactive_timeout first? (I mean there is no point of testing inactive_since if replication_slot_inactive_timeout == 0) CR4 === + if (slot->inactive_since > 0 && + replication_slot_inactive_timeout > 0) + { Same. So, instead of CR3 === and CR4 ===, I wonder if it wouldn't be better to do something like: if (replication_slot_inactive_timeout == 0) return false; else if (slot->inactive_since > 0) . . . . else return false; That would avoid checking replication_slot_inactive_timeout and inactive_since multiple times. CR5 === + * held to avoid race conditions -- for example the restart_lsn could move + * forward, or the slot could be dropped. Does the restart_lsn example makes sense here? CR6 === +static bool +InvalidateSlotForInactiveTimeout(ReplicationSlot *slot, bool need_locks) +{ InvalidatePossiblyInactiveSlot() maybe? CR7 === + /* Make sure the invalidated state persists across server restart */ + slot->just_dirtied = true; + slot->dirty = true; + SpinLockRelease(&slot->mutex); Maybe we could create a new function say MarkGivenReplicationSlotDirty() with a slot as parameter, that ReplicationSlotMarkDirty could call too? Then maybe we could set slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT in InvalidateSlotForInactiveTimeout()? (to avoid multiple SpinLockAcquire/SpinLockRelease). CR8 === + if (persist_state) + { + char path[MAXPGPATH]; + + sprintf(path, "pg_replslot/%s", NameStr(slot->data.name)); + SaveSlotToPath(slot, path, ERROR); + } Maybe we could create a new function say GivenReplicationSlotSave() with a slot as parameter, that ReplicationSlotSave() could call too? CR9 === + if (check_for_invalidation) + { + /* The slot is ours by now */ + Assert(s->active_pid == MyProcPid); + + /* + * Well, the slot is not yet ours really unless we check for the + * invalidation below. + */ + s->active_pid = 0; + if (InvalidateReplicationSlotForInactiveTimeout(s, true, true)) + { + /* + * If the slot has been invalidated, recalculate the resource + * limits. + */ + ReplicationSlotsComputeRequiredXmin(false); + ReplicationSlotsComputeRequiredLSN(); + + /* Might need it for slot clean up on error, so restore it */ + s->active_pid = MyProcPid; + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("cannot acquire invalidated replication slot \"%s\"", + NameStr(MyReplicationSlot->data.name)))); + } + s->active_pid = MyProcPid; Are we not missing some SpinLockAcquire/Release on the slot's mutex here? (the places where we set the active_pid). CR10 === @@ -1628,6 +1674,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, if (SlotIsLogical(s)) invalidation_cause = cause; break; + case RS_INVAL_INACTIVE_TIMEOUT: + if (InvalidateReplicationSlotForInactiveTimeout(s, false, false)) + invalidation_cause = cause; + break; InvalidatePossiblyObsoleteSlot() is not called with such a reason, better to use an Assert here and in the caller too? CR11 === +++ b/src/test/recovery/t/050_invalidate_slots.pl why not using 019_replslot_limit.pl? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-29 04:09 Amit Kapila <[email protected]> parent: Bharath Rupireddy <[email protected]> 3 siblings, 2 replies; 46+ messages in thread From: Amit Kapila @ 2024-03-29 04:09 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Mar 27, 2024 at 9:00 PM Bharath Rupireddy <[email protected]> wrote: > > > Thanks. I'm attaching v29 patches. 0001 managing inactive_since on the > standby for sync slots. > Commit message states: "why we can't just update inactive_since for synced slots on the standby with the value received from remote slot on the primary. This is consistent with any other slot parameter i.e. all of them are synced from the primary." The inactive_since is not consistent with other slot parameters which we copy. We don't perform anything related to those other parameters like say two_phase phase which can change that property. However, we do acquire the slot, advance the slot (as per recent discussion [1]), and release it. Since these operations can impact inactive_since, it seems to me that inactive_since is not the same as other parameters. It can have a different value than the primary. Why would anyone want to know the value of inactive_since from primary after the standby is promoted? Now, the other concern is that calling GetCurrentTimestamp() could be costly when the values for the slot are not going to be updated but if that happens we can optimize such that before acquiring the slot we can have some minimal pre-checks to ensure whether we need to update the slot or not. [1] - https://www.postgresql.org/message-id/OS0PR01MB571615D35F486080616CA841943A2%40OS0PR01MB5716.jpnprd0... -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-29 06:19 Bertrand Drouvot <[email protected]> parent: Amit Kapila <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Bertrand Drouvot @ 2024-03-29 06:19 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote: > On Wed, Mar 27, 2024 at 9:00 PM Bharath Rupireddy > <[email protected]> wrote: > > > > > > Thanks. I'm attaching v29 patches. 0001 managing inactive_since on the > > standby for sync slots. > > > > Commit message states: "why we can't just update inactive_since for > synced slots on the standby with the value received from remote slot > on the primary. This is consistent with any other slot parameter i.e. > all of them are synced from the primary." > > The inactive_since is not consistent with other slot parameters which > we copy. We don't perform anything related to those other parameters > like say two_phase phase which can change that property. However, we > do acquire the slot, advance the slot (as per recent discussion [1]), > and release it. Since these operations can impact inactive_since, it > seems to me that inactive_since is not the same as other parameters. > It can have a different value than the primary. Why would anyone want > to know the value of inactive_since from primary after the standby is > promoted? I think it can be useful "before" it is promoted and in case the primary is down. I agree that tracking the activity time of a synced slot can be useful, why not creating a dedicated field for that purpose (and keep inactive_since a perfect "copy" of the primary)? > Now, the other concern is that calling GetCurrentTimestamp() > could be costly when the values for the slot are not going to be > updated but if that happens we can optimize such that before acquiring > the slot we can have some minimal pre-checks to ensure whether we need > to update the slot or not. Right, but for a very active slot it is likely that we call GetCurrentTimestamp() during almost each sync cycle. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-29 09:33 Amit Kapila <[email protected]> parent: Bertrand Drouvot <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Amit Kapila @ 2024-03-29 09:33 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Mar 29, 2024 at 11:49 AM Bertrand Drouvot <[email protected]> wrote: > > On Fri, Mar 29, 2024 at 09:39:31AM +0530, Amit Kapila wrote: > > > > Commit message states: "why we can't just update inactive_since for > > synced slots on the standby with the value received from remote slot > > on the primary. This is consistent with any other slot parameter i.e. > > all of them are synced from the primary." > > > > The inactive_since is not consistent with other slot parameters which > > we copy. We don't perform anything related to those other parameters > > like say two_phase phase which can change that property. However, we > > do acquire the slot, advance the slot (as per recent discussion [1]), > > and release it. Since these operations can impact inactive_since, it > > seems to me that inactive_since is not the same as other parameters. > > It can have a different value than the primary. Why would anyone want > > to know the value of inactive_since from primary after the standby is > > promoted? > > I think it can be useful "before" it is promoted and in case the primary is down. > It is not clear to me what is user going to do by checking the inactivity time for slots when the corresponding server is down. I thought the idea was to check such slots and see if they need to be dropped or enabled again to avoid excessive disk usage, etc. > I agree that tracking the activity time of a synced slot can be useful, why > not creating a dedicated field for that purpose (and keep inactive_since a > perfect "copy" of the primary)? > We can have a separate field for this but not sure if it is worth it. > > Now, the other concern is that calling GetCurrentTimestamp() > > could be costly when the values for the slot are not going to be > > updated but if that happens we can optimize such that before acquiring > > the slot we can have some minimal pre-checks to ensure whether we need > > to update the slot or not. > > Right, but for a very active slot it is likely that we call GetCurrentTimestamp() > during almost each sync cycle. > True, but if we have to save a slot to disk each time to persist the changes (for an active slot) then probably GetCurrentTimestamp() shouldn't be costly enough to matter. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-03-31 04:55 Bharath Rupireddy <[email protected]> parent: Bertrand Drouvot <[email protected]> 0 siblings, 1 reply; 46+ messages in thread From: Bharath Rupireddy @ 2024-03-31 04:55 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Thu, Mar 28, 2024 at 3:13 PM Bertrand Drouvot <[email protected]> wrote: > > Regarding 0002: Thanks for reviewing it. > Some testing: > > T1 === > > When the slot is invalidated on the primary, then the reason is propagated to > the sync slot (if any). That's fine but we are loosing the inactive_since on the > standby: > > Primary: > > postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot'; > slot_name | inactive_since | conflicting | invalidation_reason > -------------+-------------------------------+-------------+--------------------- > lsub29_slot | 2024-03-28 08:24:51.672528+00 | f | inactive_timeout > (1 row) > > Standby: > > postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot'; > slot_name | inactive_since | conflicting | invalidation_reason > -------------+----------------+-------------+--------------------- > lsub29_slot | | f | inactive_timeout > (1 row) > > I think in this case it should always reflect the value from the primary (so > that one can understand why it is invalidated). I'll come back to this as soon as we all agree on inactive_since behavior for synced slots. > T2 === > > And it is set to a value during promotion: > > postgres=# select pg_promote(); > pg_promote > ------------ > t > (1 row) > > postgres=# select slot_name,inactive_since,conflicting,invalidation_reason from pg_replication_slots where slot_name='lsub29_slot'; > slot_name | inactive_since | conflicting | invalidation_reason > -------------+------------------------------+-------------+--------------------- > lsub29_slot | 2024-03-28 08:30:11.74505+00 | f | inactive_timeout > (1 row) > > I think when it is invalidated it should always reflect the value from the > primary (so that one can understand why it is invalidated). I'll come back to this as soon as we all agree on inactive_since behavior for synced slots. > T3 === > > As far the slot invalidation on the primary: > > postgres=# SELECT * FROM pg_logical_slot_get_changes('lsub29_slot', NULL, NULL, 'include-xids', '0'); > ERROR: cannot acquire invalidated replication slot "lsub29_slot" > > Can we make the message more consistent with what can be found in CreateDecodingContext() > for example? Hm, that makes sense because slot acquisition and release is something internal to the server. > T4 === > > Also, it looks like querying pg_replication_slots() does not trigger an > invalidation: I think it should if the slot is not invalidated yet (and matches > the invalidation criteria). There's a different opinion on this, check comment #3 from https://www.postgresql.org/message-id/CAA4eK1LLj%2BeaMN-K8oeOjfG%2BUuzTY%3DL5PXbcMJURZbFm%2B_aJSA%40.... > Code review: > > CR1 === > > + Invalidate replication slots that are inactive for longer than this > + amount of time. If this value is specified without units, it is taken > > s/Invalidate/Invalidates/? Done. > Should we mention the relationship with inactive_since? Done. > CR2 === > > + * > + * If check_for_invalidation is true, the slot is checked for invalidation > + * based on replication_slot_inactive_timeout GUC and an error is raised after making the slot ours. > */ > void > -ReplicationSlotAcquire(const char *name, bool nowait) > +ReplicationSlotAcquire(const char *name, bool nowait, > + bool check_for_invalidation) > > > s/check_for_invalidation/check_for_timeout_invalidation/? Done. > CR3 === > > + if (slot->inactive_since == 0 || > + replication_slot_inactive_timeout == 0) > + return false; > > Better to test replication_slot_inactive_timeout first? (I mean there is no > point of testing inactive_since if replication_slot_inactive_timeout == 0) > > CR4 === > > + if (slot->inactive_since > 0 && > + replication_slot_inactive_timeout > 0) > + { > > Same. > > So, instead of CR3 === and CR4 ===, I wonder if it wouldn't be better to do > something like: > > if (replication_slot_inactive_timeout == 0) > return false; > else if (slot->inactive_since > 0) > . > else > return false; > > That would avoid checking replication_slot_inactive_timeout and inactive_since > multiple times. Done. > CR5 === > > + * held to avoid race conditions -- for example the restart_lsn could move > + * forward, or the slot could be dropped. > > Does the restart_lsn example makes sense here? No, it doesn't. Modified that. > CR6 === > > +static bool > +InvalidateSlotForInactiveTimeout(ReplicationSlot *slot, bool need_locks) > +{ > > InvalidatePossiblyInactiveSlot() maybe? I think we will lose the essence i.e. timeout from the suggested function name, otherwise just the inactive doesn't give a clearer meaning. I kept it that way unless anyone suggests otherwise. > CR7 === > > + /* Make sure the invalidated state persists across server restart */ > + slot->just_dirtied = true; > + slot->dirty = true; > + SpinLockRelease(&slot->mutex); > > Maybe we could create a new function say MarkGivenReplicationSlotDirty() > with a slot as parameter, that ReplicationSlotMarkDirty could call too? Done that. > Then maybe we could set slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT in > InvalidateSlotForInactiveTimeout()? (to avoid multiple SpinLockAcquire/SpinLockRelease). Done that. > CR8 === > > + if (persist_state) > + { > + char path[MAXPGPATH]; > + > + sprintf(path, "pg_replslot/%s", NameStr(slot->data.name)); > + SaveSlotToPath(slot, path, ERROR); > + } > > Maybe we could create a new function say GivenReplicationSlotSave() > with a slot as parameter, that ReplicationSlotSave() could call too? Done that. > CR9 === > > + if (check_for_invalidation) > + { > + /* The slot is ours by now */ > + Assert(s->active_pid == MyProcPid); > + > + /* > + * Well, the slot is not yet ours really unless we check for the > + * invalidation below. > + */ > + s->active_pid = 0; > + if (InvalidateReplicationSlotForInactiveTimeout(s, true, true)) > + { > + /* > + * If the slot has been invalidated, recalculate the resource > + * limits. > + */ > + ReplicationSlotsComputeRequiredXmin(false); > + ReplicationSlotsComputeRequiredLSN(); > + > + /* Might need it for slot clean up on error, so restore it */ > + s->active_pid = MyProcPid; > + ereport(ERROR, > + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), > + errmsg("cannot acquire invalidated replication slot \"%s\"", > + NameStr(MyReplicationSlot->data.name)))); > + } > + s->active_pid = MyProcPid; > > Are we not missing some SpinLockAcquire/Release on the slot's mutex here? (the > places where we set the active_pid). Hm, yes. But, shall I acquire the mutex, set active_pid to 0 for a moment just to satisfy Assert(slot->active_pid == 0); in InvalidateReplicationSlotForInactiveTimeout and InvalidateSlotForInactiveTimeout? I just removed the assertions because being replication_slot_inactive_timeout > 0 and inactive_since > 0 is enough for these functions to think and decide on inactive timeout invalidation. > CR10 === > > @@ -1628,6 +1674,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, > if (SlotIsLogical(s)) > invalidation_cause = cause; > break; > + case RS_INVAL_INACTIVE_TIMEOUT: > + if (InvalidateReplicationSlotForInactiveTimeout(s, false, false)) > + invalidation_cause = cause; > + break; > > InvalidatePossiblyObsoleteSlot() is not called with such a reason, better to use > an Assert here and in the caller too? Done. > CR11 === > > +++ b/src/test/recovery/t/050_invalidate_slots.pl > > why not using 019_replslot_limit.pl? I understand that 019_replslot_limit covers wal_removed related invalidations. But, I don't want to kludge it with a bunch of other tests. The new tests anyway need a bunch of new nodes and a couple of helper functions. Any future invalidation mechanisms can be added here in this new file. Also, having a separate file quickly helps isolate any test failures that BF animals might report in future. I don't think a separate test file here hurts anyone unless there's a strong reason against it. Please see the attached v30 patch. 0002 is where all of the above review comments have been addressed. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com Attachments: [application/x-patch] v30-0001-Maintain-inactive_since-for-synced-slots-correct.patch (16.5K, ../../CALj2ACVVb1Uds+ucw_nq03uoGB5NToVi5hSiwW+S_goju_sncg@mail.gmail.com/2-v30-0001-Maintain-inactive_since-for-synced-slots-correct.patch) download | inline diff: From b84cf9dc5d20e202e08c372e0aa7850966ed7271 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Sat, 30 Mar 2024 20:52:48 +0000 Subject: [PATCH v30 1/2] Maintain inactive_since for synced slots correctly. The slot's inactive_since isn't currently maintained for synced slots on the standby. The commit a11f330b55 prevents updating inactive_since with RecoveryInProgress() check in RestoreSlotFromDisk(). But, the issue is that RecoveryInProgress() always returns true in RestoreSlotFromDisk() as 'xlogctl->SharedRecoveryState' is always 'RECOVERY_STATE_CRASH' at that time. Because of this, inactive_since is always NULL on a promoted standby for all synced slots even after server restart. Above issue led us to a question as to why we can't just update inactive_since for synced slots on the standby with the value received from remote slot on the primary. This is consistent with any other slot parameter i.e. all of them are synced from the primary. This commit does two things: 1) Updates inactive_since for sync slots with the value received from the primary's slot. 2) Ensures the value is set to current timestamp during the shutdown of slot sync machinery to help correctly interpret the time if the standby gets promoted without a restart. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACWLctoiH-pSjWnEpR54q4DED6rw_BRJm5pCx86_Y01MoQ%40mail.gmail.com --- --- doc/src/sgml/system-views.sgml | 9 +++ src/backend/replication/logical/slotsync.c | 61 ++++++++++++++++++- src/backend/replication/slot.c | 40 ++++++++---- src/test/perl/PostgreSQL/Test/Cluster.pm | 34 +++++++++++ src/test/recovery/t/019_replslot_limit.pl | 26 +------- .../t/040_standby_failover_slots_sync.pl | 43 +++++++++++++ 6 files changed, 173 insertions(+), 40 deletions(-) diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 3c8dca8ca3..c8d97ab375 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2530,6 +2530,15 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <para> The time since the slot has become inactive. <literal>NULL</literal> if the slot is currently being used. + Note that the slots on the standbys that are being synced from a + primary server (whose <structfield>synced</structfield> field is + <literal>true</literal>), will get the + <structfield>inactive_since</structfield> value from the + corresponding remote slot on the primary. Also, after the standby + starts up, the <structfield>inactive_since</structfield> value + (for such synced slots) will remain <literal>NULL</literal> until + the next synchronization (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>). </para></entry> </row> diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 30480960c5..9c95a4b062 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -137,9 +137,12 @@ typedef struct RemoteSlot /* RS_INVAL_NONE if valid, or the reason of invalidation */ ReplicationSlotInvalidationCause invalidated; + + TimestampTz inactive_since; /* in seconds */ } RemoteSlot; static void slotsync_failure_callback(int code, Datum arg); +static void update_synced_slots_inactive_since(void); /* * If necessary, update the local synced slot's metadata based on the data @@ -167,6 +170,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) remote_slot->two_phase == slot->data.two_phase && remote_slot->failover == slot->data.failover && remote_slot->confirmed_lsn == slot->data.confirmed_flush && + remote_slot->inactive_since == slot->inactive_since && strcmp(remote_slot->plugin, NameStr(slot->data.plugin)) == 0) return false; @@ -182,6 +186,7 @@ update_local_synced_slot(RemoteSlot *remote_slot, Oid remote_dbid) slot->data.confirmed_flush = remote_slot->confirmed_lsn; slot->data.catalog_xmin = remote_slot->catalog_xmin; slot->effective_catalog_xmin = remote_slot->catalog_xmin; + slot->inactive_since = remote_slot->inactive_since; SpinLockRelease(&slot->mutex); if (xmin_changed) @@ -652,9 +657,9 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) static bool synchronize_slots(WalReceiverConn *wrconn) { -#define SLOTSYNC_COLUMN_COUNT 9 +#define SLOTSYNC_COLUMN_COUNT 10 Oid slotRow[SLOTSYNC_COLUMN_COUNT] = {TEXTOID, TEXTOID, LSNOID, - LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID}; + LSNOID, XIDOID, BOOLOID, BOOLOID, TEXTOID, TEXTOID, TIMESTAMPTZOID}; WalRcvExecResult *res; TupleTableSlot *tupslot; @@ -663,7 +668,7 @@ synchronize_slots(WalReceiverConn *wrconn) bool started_tx = false; const char *query = "SELECT slot_name, plugin, confirmed_flush_lsn," " restart_lsn, catalog_xmin, two_phase, failover," - " database, invalidation_reason" + " database, invalidation_reason, inactive_since" " FROM pg_catalog.pg_replication_slots" " WHERE failover and NOT temporary"; @@ -743,6 +748,13 @@ synchronize_slots(WalReceiverConn *wrconn) remote_slot->invalidated = isnull ? RS_INVAL_NONE : GetSlotInvalidationCause(TextDatumGetCString(d)); + /* + * It is possible to get null value for inactive_since if the slot is + * active on the primary server, so handle accordingly. + */ + d = slot_getattr(tupslot, ++col, &isnull); + remote_slot->inactive_since = isnull ? 0 : DatumGetTimestampTz(d); + /* Sanity check */ Assert(col == SLOTSYNC_COLUMN_COUNT); @@ -1296,6 +1308,46 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len) Assert(false); } +/* + * Update the inactive_since property for synced slots. + */ +static void +update_synced_slots_inactive_since(void) +{ + TimestampTz now = 0; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + + /* + * We get the current time beforehand and only once to avoid + * system calls overhead while holding the lock. + */ + if (now == 0) + now = GetCurrentTimestamp(); + + /* + * Set the time since the slot has become inactive after shutting + * down slot sync machinery. This helps correctly interpret the + * time if the standby gets promoted without a restart. + */ + SpinLockAcquire(&s->mutex); + s->inactive_since = now; + SpinLockRelease(&s->mutex); + } + } + + LWLockRelease(ReplicationSlotControlLock); +} + /* * Shut down the slot sync worker. */ @@ -1309,6 +1361,7 @@ ShutDownSlotSync(void) if (SlotSyncCtx->pid == InvalidPid) { SpinLockRelease(&SlotSyncCtx->mutex); + update_synced_slots_inactive_since(); return; } SpinLockRelease(&SlotSyncCtx->mutex); @@ -1341,6 +1394,8 @@ ShutDownSlotSync(void) } SpinLockRelease(&SlotSyncCtx->mutex); + + update_synced_slots_inactive_since(); } /* diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index d778c0b921..7dbb44b7b0 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -42,6 +42,7 @@ #include "access/transam.h" #include "access/xlog_internal.h" #include "access/xlogrecovery.h" +#include "access/xlogutils.h" #include "common/file_utils.h" #include "common/string.h" #include "miscadmin.h" @@ -655,6 +656,7 @@ ReplicationSlotRelease(void) char *slotname = NULL; /* keep compiler quiet */ bool is_logical = false; /* keep compiler quiet */ TimestampTz now = 0; + bool is_slot_being_synced = false; Assert(slot != NULL && slot->active_pid != 0); @@ -690,12 +692,15 @@ ReplicationSlotRelease(void) } /* - * Set the last inactive time after marking the slot inactive. We don't - * set it for the slots currently being synced from the primary to the - * standby because such slots are typically inactive as decoding is not - * allowed on those. + * Set the time since the slot has become inactive. + * + * Note that we don't set it for the slots currently being synced from the + * primary to the standby, because such slots typically sync the data from + * the remote slot. */ - if (!(RecoveryInProgress() && slot->data.synced)) + if (RecoveryInProgress() && slot->data.synced) + is_slot_being_synced = true; + else now = GetCurrentTimestamp(); if (slot->data.persistency == RS_PERSISTENT) @@ -706,11 +711,12 @@ ReplicationSlotRelease(void) */ SpinLockAcquire(&slot->mutex); slot->active_pid = 0; - slot->inactive_since = now; + if (!is_slot_being_synced) + slot->inactive_since = now; SpinLockRelease(&slot->mutex); ConditionVariableBroadcast(&slot->active_cv); } - else + else if (!is_slot_being_synced) { SpinLockAcquire(&slot->mutex); slot->inactive_since = now; @@ -2369,13 +2375,21 @@ RestoreSlotFromDisk(const char *name) slot->active_pid = 0; /* - * We set the last inactive time after loading the slot from the disk - * into memory. Whoever acquires the slot i.e. makes the slot active - * will reset it. We don't set it for the slots currently being synced - * from the primary to the standby because such slots are typically - * inactive as decoding is not allowed on those. + * Set the time since the slot has become inactive after loading the + * slot from the disk into memory. Whoever acquires the slot i.e. + * makes the slot active will reset it. + * + * Note that we don't set it for the slots currently being synced from + * the primary to the standby, because such slots typically sync the + * data from the remote slot. We use InRecovery flag instead of + * RecoveryInProgress() as the latter always returns true at this time + * even on primary. + * + * Note that for synced slots after the standby starts up (i.e. after + * the slots are loaded from the disk), the inactive_since will remain + * zero until the next slot sync cycle. */ - if (!(RecoveryInProgress() && slot->data.synced)) + if (!(InRecovery && slot->data.synced)) slot->inactive_since = GetCurrentTimestamp(); else slot->inactive_since = 0; diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index b08296605c..ddfc3236f3 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -3276,6 +3276,40 @@ sub create_logical_slot_on_standby =pod +=item $node->get_slot_inactive_since_value(self, slot_name, reference_time) + +Get inactive_since column value for a given replication slot validating it +against optional reference time. + +=cut + +sub get_slot_inactive_since_value +{ + my ($self, $slot_name, $reference_time) = @_; + my $name = $self->name; + + my $inactive_since = $self->safe_psql('postgres', + qq(SELECT inactive_since FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) + ); + + # Check that the captured time is sane + if (defined $reference_time) + { + is($self->safe_psql('postgres', + qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND + '$inactive_since'::timestamptz >= '$reference_time'::timestamptz;] + ), + 't', + "last inactive time for slot $slot_name is valid on node $name") + or die "could not validate captured inactive_since for slot $slot_name"; + } + + return $inactive_since; +} + +=pod + =item $node->advance_wal(num) Advance WAL of node by given number of segments. diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl index 3b9a306a8b..c8e5e5054d 100644 --- a/src/test/recovery/t/019_replslot_limit.pl +++ b/src/test/recovery/t/019_replslot_limit.pl @@ -443,7 +443,7 @@ $primary4->safe_psql( # Get inactive_since value after the slot's creation. Note that the slot is # still inactive till it's used by the standby below. my $inactive_since = - capture_and_validate_slot_inactive_since($primary4, $sb4_slot, $slot_creation_time); + $primary4->get_slot_inactive_since_value($sb4_slot, $slot_creation_time); $standby4->start; @@ -502,7 +502,7 @@ $publisher4->safe_psql('postgres', # Get inactive_since value after the slot's creation. Note that the slot is # still inactive till it's used by the subscriber below. $inactive_since = - capture_and_validate_slot_inactive_since($publisher4, $lsub4_slot, $slot_creation_time); + $publisher4->get_slot_inactive_since_value($lsub4_slot, $slot_creation_time); $subscriber4->start; $subscriber4->safe_psql('postgres', @@ -540,26 +540,4 @@ is( $publisher4->safe_psql( $publisher4->stop; $subscriber4->stop; -# Capture and validate inactive_since of a given slot. -sub capture_and_validate_slot_inactive_since -{ - my ($node, $slot_name, $slot_creation_time) = @_; - - my $inactive_since = $node->safe_psql('postgres', - qq(SELECT inactive_since FROM pg_replication_slots - WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) - ); - - # Check that the captured time is sane - is( $node->safe_psql( - 'postgres', - qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND - '$inactive_since'::timestamptz >= '$slot_creation_time'::timestamptz;] - ), - 't', - "last inactive time for an active slot $slot_name is sane"); - - return $inactive_since; -} - done_testing(); diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index f47bfd78eb..33e3a8dcf0 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1'); $subscriber1->init; $subscriber1->start; +# Capture the time before the logical failover slot is created on the +# primary. We later call this publisher as primary anyway. +my $slot_creation_time_on_primary = $publisher->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + # Create a slot on the publisher with failover disabled $publisher->safe_psql('postgres', "SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);" @@ -174,6 +181,10 @@ $primary->poll_query_until( "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", 1); +# Capture the inactive_since of the slot from the primary +my $inactive_since_on_primary = + $primary->get_slot_inactive_since_value('lsub1_slot', $slot_creation_time_on_primary); + # Wait for the standby to catch up so that the standby is not lagging behind # the subscriber. $primary->wait_for_replay_catchup($standby1); @@ -190,6 +201,18 @@ is( $standby1->safe_psql( "t", 'logical slots have synced as true on standby'); +# Capture the inactive_since of the synced slot on the standby +my $inactive_since_on_standby = + $standby1->get_slot_inactive_since_value('lsub1_slot', $slot_creation_time_on_primary); + +# Synced slots on the standby must get the inactive_since from the primary. +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_primary'::timestamptz = '$inactive_since_on_standby'::timestamptz;" + ), + "t", + 'synchronized slot has got the inactive_since from the primary'); + ################################################## # Test that the synchronized slot will be dropped if the corresponding remote # slot on the primary server has been dropped. @@ -750,8 +773,28 @@ $primary->reload; $standby1->start; $primary->wait_for_replay_catchup($standby1); +# Capture the time before the standby is promoted +my $promotion_time_on_primary = $standby1->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + $standby1->promote; +# Capture the inactive_since of the synced slot after the promotion. +# Expectation here is that the slot gets its own inactive_since as part of the +# promotion. We do this check before the slot is enabled on the new primary +# below, otherwise the slot gets active setting inactive_since to NULL. +my $inactive_since_on_new_primary = + $standby1->get_slot_inactive_since_value('lsub1_slot', $promotion_time_on_primary); + +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_new_primary'::timestamptz > '$inactive_since_on_primary'::timestamptz" + ), + "t", + 'synchronized slot has got its own inactive_since on the new primary'); + # Update subscription with the new primary's connection info my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; $subscriber1->safe_psql('postgres', -- 2.34.1 [application/x-patch] v30-0002-Add-inactive_timeout-based-replication-slot-inva.patch (31.5K, ../../CALj2ACVVb1Uds+ucw_nq03uoGB5NToVi5hSiwW+S_goju_sncg@mail.gmail.com/3-v30-0002-Add-inactive_timeout-based-replication-slot-inva.patch) download | inline diff: From 64bb4f8396595dae62ee07726dfacadb6e87f119 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Sun, 31 Mar 2024 04:40:42 +0000 Subject: [PATCH v30 2/2] Add inactive_timeout based replication slot invalidation. Till now, postgres has the ability to invalidate inactive replication slots based on the amount of WAL (set via max_slot_wal_keep_size GUC) that will be needed for the slots in case they become active. However, choosing a default value for max_slot_wal_keep_size is tricky. Because the amount of WAL a customer generates, and their allocated storage will vary greatly in production, making it difficult to pin down a one-size-fits-all value. It is often easy for developers to set a timeout of say 1 or 2 or 3 days at slot level, after which the inactive slots get dropped. To achieve the above, postgres introduces a GUC allowing users set inactive timeout and then a slot stays inactive for this much amount of time it invalidates the slot. The invalidation check happens at various locations to help being as latest as possible, these locations include the following: - Whenever the slot is acquired and the slot acquisition errors out if invalidated. - During checkpoint Note that this new invalidation mechanism won't kick-in for the slots that are currently being synced from the primary to the standby. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com --- doc/src/sgml/config.sgml | 25 ++ doc/src/sgml/system-views.sgml | 7 + .../replication/logical/logicalfuncs.c | 2 +- src/backend/replication/logical/slotsync.c | 4 +- src/backend/replication/slot.c | 196 +++++++++++- src/backend/replication/slotfuncs.c | 2 +- src/backend/replication/walsender.c | 4 +- src/backend/utils/adt/pg_upgrade_support.c | 2 +- src/backend/utils/misc/guc_tables.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/replication/slot.h | 8 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/050_invalidate_slots.pl | 278 ++++++++++++++++++ 13 files changed, 518 insertions(+), 24 deletions(-) create mode 100644 src/test/recovery/t/050_invalidate_slots.pl diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index f65c17e5ae..126b461bb1 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4545,6 +4545,31 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows </listitem> </varlistentry> + <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout"> + <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>) + <indexterm> + <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Invalidates replication slots that are inactive for longer the + specified amount of time. If this value is specified without units, + it is taken as seconds. A value of zero (which is default) disables + the timeout mechanism. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server + command line. + </para> + + <para> + The timeout is measured from the time since the slot has become + inactive (known from its + <structfield>inactive_since</structfield> value) until it gets + used (i.e., its <structfield>active</structfield> is set to true). + </para> + </listitem> + </varlistentry> + <varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp"> <term><varname>track_commit_timestamp</varname> (<type>boolean</type>) <indexterm> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index c8d97ab375..5c05fd1c07 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2582,6 +2582,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx perform logical decoding. It is set only for logical slots. </para> </listitem> + <listitem> + <para> + <literal>inactive_timeout</literal> means that the slot has been + inactive for the duration specified by + <xref linkend="guc-replication-slot-inactive-timeout"/> parameter. + </para> + </listitem> </itemizedlist> </para></entry> </row> diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c index b4dd5cce75..56fc1a45a9 100644 --- a/src/backend/replication/logical/logicalfuncs.c +++ b/src/backend/replication/logical/logicalfuncs.c @@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin else end_of_wal = GetXLogReplayRecPtr(NULL); - ReplicationSlotAcquire(NameStr(*name), true); + ReplicationSlotAcquire(NameStr(*name), true, true); PG_TRY(); { diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 9c95a4b062..71b6a254cf 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -321,7 +321,7 @@ drop_local_obsolete_slots(List *remote_slot_list) if (synced_slot) { - ReplicationSlotAcquire(NameStr(local_slot->data.name), true); + ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false); ReplicationSlotDropAcquired(); } @@ -531,7 +531,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) * InvalidatePossiblyObsoleteSlot() where it invalidates slot directly * if the slot is not acquired by other processes. */ - ReplicationSlotAcquire(remote_slot->name, true); + ReplicationSlotAcquire(remote_slot->name, true, false); Assert(slot == MyReplicationSlot); diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 7dbb44b7b0..7182d89b58 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -108,10 +108,11 @@ const char *const SlotInvalidationCauses[] = { [RS_INVAL_WAL_REMOVED] = "wal_removed", [RS_INVAL_HORIZON] = "rows_removed", [RS_INVAL_WAL_LEVEL] = "wal_level_insufficient", + [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout", }; /* Maximum number of invalidation causes */ -#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL +#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1), "array length mismatch"); @@ -141,6 +142,7 @@ ReplicationSlot *MyReplicationSlot = NULL; /* GUC variables */ int max_replication_slots = 10; /* the maximum number of replication * slots */ +int replication_slot_inactive_timeout = 0; /* * This GUC lists streaming replication standby server slot names that @@ -159,6 +161,7 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr; static void ReplicationSlotShmemExit(int code, Datum arg); static void ReplicationSlotDropPtr(ReplicationSlot *slot); +static bool InvalidateSlotForInactiveTimeout(ReplicationSlot *slot); /* internal persistency functions */ static void RestoreSlotFromDisk(const char *name); @@ -536,9 +539,14 @@ ReplicationSlotName(int index, Name name) * * An error is raised if nowait is true and the slot is currently in use. If * nowait is false, we sleep until the slot is released by the owning process. + * + * If check_for_timeout_invalidation is true, the slot is checked for + * invalidation based on replication_slot_inactive_timeout GUC, and an error is + * raised after making the slot ours. */ void -ReplicationSlotAcquire(const char *name, bool nowait) +ReplicationSlotAcquire(const char *name, bool nowait, + bool check_for_timeout_invalidation) { ReplicationSlot *s; int active_pid; @@ -616,6 +624,34 @@ retry: /* We made this slot active, so it's ours now. */ MyReplicationSlot = s; + /* + * Check if the given slot can be invalidated based on its inactive + * timeout. If yes, persist the invalidated state to disk and then error + * out. We do this only after making the slot ours to avoid anyone else + * acquiring it while we check for its invalidation. + */ + if (check_for_timeout_invalidation) + { + /* The slot is ours by now */ + Assert(s->active_pid == MyProcPid); + + if (InvalidateReplicationSlotForInactiveTimeout(s, true)) + { + /* + * If the slot has been invalidated, recalculate the resource + * limits. + */ + ReplicationSlotsComputeRequiredXmin(false); + ReplicationSlotsComputeRequiredLSN(); + + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("can no longer get changes from replication slot \"%s\"", + NameStr(MyReplicationSlot->data.name)), + errdetail("This slot has been invalidated because it was inactive for more than the time specified by replication_slot_inactive_timeout parameter."))); + } + } + /* * The call to pgstat_acquire_replslot() protects against stats for a * different slot, from before a restart or such, being present during @@ -790,7 +826,7 @@ ReplicationSlotDrop(const char *name, bool nowait) { Assert(MyReplicationSlot == NULL); - ReplicationSlotAcquire(name, nowait); + ReplicationSlotAcquire(name, nowait, false); /* * Do not allow users to drop the slots which are currently being synced @@ -813,7 +849,7 @@ ReplicationSlotAlter(const char *name, bool failover) { Assert(MyReplicationSlot == NULL); - ReplicationSlotAcquire(name, false); + ReplicationSlotAcquire(name, false, true); if (SlotIsPhysical(MyReplicationSlot)) ereport(ERROR, @@ -989,6 +1025,20 @@ ReplicationSlotDropPtr(ReplicationSlot *slot) LWLockRelease(ReplicationSlotAllocationLock); } +/* + * Helper for ReplicationSlotSave + */ +static inline void +SaveGivenReplicationSlot(ReplicationSlot *slot, int elevel) +{ + char path[MAXPGPATH]; + + Assert(slot != NULL); + + sprintf(path, "pg_replslot/%s", NameStr(slot->data.name)); + SaveSlotToPath(slot, path, elevel); +} + /* * Serialize the currently acquired slot's state from memory to disk, thereby * guaranteeing the current state will survive a crash. @@ -996,12 +1046,21 @@ ReplicationSlotDropPtr(ReplicationSlot *slot) void ReplicationSlotSave(void) { - char path[MAXPGPATH]; + SaveGivenReplicationSlot(MyReplicationSlot, ERROR); +} - Assert(MyReplicationSlot != NULL); +/* + * Helper for ReplicationSlotMarkDirty + */ +static inline void +MarkGivenReplicationSlotDirty(ReplicationSlot *slot) +{ + Assert(slot != NULL); - sprintf(path, "pg_replslot/%s", NameStr(MyReplicationSlot->data.name)); - SaveSlotToPath(MyReplicationSlot, path, ERROR); + SpinLockAcquire(&slot->mutex); + slot->just_dirtied = true; + slot->dirty = true; + SpinLockRelease(&slot->mutex); } /* @@ -1014,14 +1073,7 @@ ReplicationSlotSave(void) void ReplicationSlotMarkDirty(void) { - ReplicationSlot *slot = MyReplicationSlot; - - Assert(MyReplicationSlot != NULL); - - SpinLockAcquire(&slot->mutex); - MyReplicationSlot->just_dirtied = true; - MyReplicationSlot->dirty = true; - SpinLockRelease(&slot->mutex); + MarkGivenReplicationSlotDirty(MyReplicationSlot); } /* @@ -1515,6 +1567,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause, case RS_INVAL_WAL_LEVEL: appendStringInfoString(&err_detail, _("Logical decoding on standby requires wal_level >= logical on the primary server.")); break; + case RS_INVAL_INACTIVE_TIMEOUT: + appendStringInfoString(&err_detail, _("The slot has been inactive for more than the time specified by replication_slot_inactive_timeout parameter.")); + break; case RS_INVAL_NONE: pg_unreachable(); } @@ -1559,6 +1614,13 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr; ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE; + /* + * This function isn't expected to be called for inactive timeout based + * invalidation. A separate function + * InvalidateReplicationSlotForInactiveTimeout is to be used for that. + */ + Assert(cause != RS_INVAL_INACTIVE_TIMEOUT); + for (;;) { XLogRecPtr restart_lsn; @@ -1628,6 +1690,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, if (SlotIsLogical(s)) invalidation_cause = cause; break; + case RS_INVAL_INACTIVE_TIMEOUT: + /* not reachable */ + Assert(false); + break; case RS_INVAL_NONE: pg_unreachable(); } @@ -1781,6 +1847,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, * - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given * db; dboid may be InvalidOid for shared relations * - RS_INVAL_WAL_LEVEL: is logical + * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout occurs * * NB - this runs as part of checkpoint, so avoid raising errors if possible. */ @@ -1796,6 +1863,13 @@ InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause cause, Assert(cause != RS_INVAL_WAL_REMOVED || oldestSegno > 0); Assert(cause != RS_INVAL_NONE); + /* + * This function isn't expected to be called for inactive timeout based + * invalidation. A separate function + * InvalidateReplicationSlotForInactiveTimeout is to be used for that. + */ + Assert(cause != RS_INVAL_INACTIVE_TIMEOUT); + if (max_replication_slots == 0) return invalidated; @@ -1832,6 +1906,81 @@ restart: return invalidated; } +/* + * Invalidate given slot based on replication_slot_inactive_timeout GUC. + * + * Returns true if the slot has got invalidated. + * + * NB - this function also runs as part of checkpoint, so avoid raising errors + * if possible. + */ +bool +InvalidateReplicationSlotForInactiveTimeout(ReplicationSlot *slot, + bool persist_state) +{ + if (!InvalidateSlotForInactiveTimeout(slot)) + return false; + + /* Make sure the invalidated state persists across server restart */ + MarkGivenReplicationSlotDirty(slot); + + if (persist_state) + SaveGivenReplicationSlot(slot, ERROR); + + ReportSlotInvalidation(RS_INVAL_INACTIVE_TIMEOUT, false, 0, + slot->data.name, InvalidXLogRecPtr, + InvalidXLogRecPtr, InvalidTransactionId); + + return true; +} + +/* + * Helper for InvalidateReplicationSlotForInactiveTimeout + */ +static bool +InvalidateSlotForInactiveTimeout(ReplicationSlot *slot) +{ + ReplicationSlotInvalidationCause inavidation_cause = RS_INVAL_NONE; + + if (replication_slot_inactive_timeout == 0) + return false; + else if (slot->inactive_since > 0) + { + TimestampTz now; + + /* + * Do not invalidate the slots which are currently being synced from + * the primary to the standby. + */ + if (RecoveryInProgress() && slot->data.synced) + return false; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + SpinLockAcquire(&slot->mutex); + + /* + * Check if the slot needs to be invalidated due to + * replication_slot_inactive_timeout GUC. We do this with the spinlock + * held to avoid race conditions -- for example the inactive_since + * could change, or the slot could be dropped. + */ + now = GetCurrentTimestamp(); + if (TimestampDifferenceExceeds(slot->inactive_since, now, + replication_slot_inactive_timeout * 1000)) + { + inavidation_cause = RS_INVAL_INACTIVE_TIMEOUT; + slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT; + } + + SpinLockRelease(&slot->mutex); + LWLockRelease(ReplicationSlotControlLock); + + return (inavidation_cause == RS_INVAL_INACTIVE_TIMEOUT); + } + + return false; +} + /* * Flush all replication slots to disk. * @@ -1844,6 +1993,7 @@ void CheckPointReplicationSlots(bool is_shutdown) { int i; + bool invalidated = false; elog(DEBUG1, "performing replication slot checkpoint"); @@ -1867,6 +2017,13 @@ CheckPointReplicationSlots(bool is_shutdown) /* save the slot to disk, locking is handled in SaveSlotToPath() */ sprintf(path, "pg_replslot/%s", NameStr(s->data.name)); + /* + * Here's an opportunity to invalidate inactive replication slots + * based on timeout, so let's do it. + */ + if (InvalidateReplicationSlotForInactiveTimeout(s, false)) + invalidated = true; + /* * Slot's data is not flushed each time the confirmed_flush LSN is * updated as that could lead to frequent writes. However, we decide @@ -1893,6 +2050,13 @@ CheckPointReplicationSlots(bool is_shutdown) SaveSlotToPath(s, path, LOG); } LWLockRelease(ReplicationSlotAllocationLock); + + /* If the slot has been invalidated, recalculate the resource limits */ + if (invalidated) + { + ReplicationSlotsComputeRequiredXmin(false); + ReplicationSlotsComputeRequiredLSN(); + } } /* diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index da57177c25..677c0bf0a2 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -651,7 +651,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS) moveto = Min(moveto, GetXLogReplayRecPtr(NULL)); /* Acquire the slot so we "own" it */ - ReplicationSlotAcquire(NameStr(*slotname), true); + ReplicationSlotAcquire(NameStr(*slotname), true, true); /* A slot whose restart_lsn has never been reserved cannot be advanced */ if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn)) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index bc40c454de..96eeb8b7d2 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd) if (cmd->slotname) { - ReplicationSlotAcquire(cmd->slotname, true); + ReplicationSlotAcquire(cmd->slotname, true, true); if (SlotIsLogical(MyReplicationSlot)) ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), @@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) Assert(!MyReplicationSlot); - ReplicationSlotAcquire(cmd->slotname, true); + ReplicationSlotAcquire(cmd->slotname, true, true); /* * Force a disconnect, so that the decoding code doesn't need to care diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index c54b08fe18..82956d58d3 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS) slot_name = PG_GETARG_NAME(0); /* Acquire the given slot */ - ReplicationSlotAcquire(NameStr(*slot_name), true); + ReplicationSlotAcquire(NameStr(*slot_name), true, false); Assert(SlotIsLogical(MyReplicationSlot)); diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 92fcd5fa4d..c63f76505f 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -2971,6 +2971,18 @@ struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING, + gettext_noop("Sets the amount of time to wait before invalidating an " + "inactive replication slot."), + NULL, + GUC_UNIT_S + }, + &replication_slot_inactive_timeout, + 0, 0, INT_MAX, + NULL, NULL, NULL + }, + { {"commit_delay", PGC_SUSET, WAL_SETTINGS, gettext_noop("Sets the delay in microseconds between transaction commit and " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index adcc0257f9..18dd57e589 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -334,6 +334,7 @@ #wal_sender_timeout = 60s # in milliseconds; 0 disables #track_commit_timestamp = off # collect timestamp of transaction commit # (change requires restart) +#replication_slot_inactive_timeout = 0 # in seconds; 0 disables # - Primary Server - diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index 7b937d1a0c..8f5e602745 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause RS_INVAL_HORIZON, /* wal_level insufficient for slot */ RS_INVAL_WAL_LEVEL, + /* inactive slot timeout has occurred */ + RS_INVAL_INACTIVE_TIMEOUT, } ReplicationSlotInvalidationCause; extern PGDLLIMPORT const char *const SlotInvalidationCauses[]; @@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot; /* GUCs */ extern PGDLLIMPORT int max_replication_slots; extern PGDLLIMPORT char *standby_slot_names; +extern PGDLLIMPORT int replication_slot_inactive_timeout; /* shmem initialization functions */ extern Size ReplicationSlotsShmemSize(void); @@ -245,7 +248,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait); extern void ReplicationSlotDropAcquired(void); extern void ReplicationSlotAlter(const char *name, bool failover); -extern void ReplicationSlotAcquire(const char *name, bool nowait); +extern void ReplicationSlotAcquire(const char *name, bool nowait, + bool check_for_timeout_invalidation); extern void ReplicationSlotRelease(void); extern void ReplicationSlotCleanup(void); extern void ReplicationSlotSave(void); @@ -264,6 +268,8 @@ extern bool InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause XLogSegNo oldestSegno, Oid dboid, TransactionId snapshotConflictHorizon); +extern bool InvalidateReplicationSlotForInactiveTimeout(ReplicationSlot *slot, + bool persist_state); extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_lock); extern int ReplicationSlotIndex(ReplicationSlot *slot); extern bool ReplicationSlotName(int index, Name name); diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index b1eb77b1ec..708a2a3798 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -51,6 +51,7 @@ tests += { 't/040_standby_failover_slots_sync.pl', 't/041_checkpoint_at_promote.pl', 't/042_low_level_backup.pl', + 't/050_invalidate_slots.pl', ], }, } diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl new file mode 100644 index 0000000000..8e919915f1 --- /dev/null +++ b/src/test/recovery/t/050_invalidate_slots.pl @@ -0,0 +1,278 @@ +# Copyright (c) 2024, PostgreSQL Global Development Group + +# Test for replication slots invalidation +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Utils; +use PostgreSQL::Test::Cluster; +use Test::More; +use Time::HiRes qw(usleep); + +# ============================================================================= +# Testcase start: Invalidate streaming standby's slot as well as logical +# failover slot on primary due to inactive timeout GUC. Also, check the logical +# failover slot synced on to the standby doesn't invalidate the slot on its own, +# but gets the invalidated state from the remote slot on the primary. + +# Initialize primary node +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init(allows_streaming => 'logical'); + +# Avoid checkpoint during the test, otherwise, the test can get unpredictable +$primary->append_conf( + 'postgresql.conf', q{ +checkpoint_timeout = 1h +autovacuum = off +}); +$primary->start; + +# Take backup +my $backup_name = 'my_backup'; +$primary->backup($backup_name); + +# Create a standby linking to the primary using the replication slot +my $standby1 = PostgreSQL::Test::Cluster->new('standby1'); +$standby1->init_from_backup($primary, $backup_name, has_streaming => 1); + +my $connstr_1 = $primary->connstr; +$standby1->append_conf( + 'postgresql.conf', qq( +hot_standby_feedback = on +primary_slot_name = 'sb1_slot' +primary_conninfo = '$connstr_1 dbname=postgres' +)); + +# Create sync slot on the primary +$primary->psql('postgres', + q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);} +); + +$primary->safe_psql( + 'postgres', qq[ + SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot'); +]); + +$standby1->start; + +my $standby1_logstart = -s $standby1->logfile; + +# Wait until standby has replayed enough data +$primary->wait_for_catchup($standby1); + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the logical failover slot is created on the standby and is +# flagged as 'synced'. +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 1 FROM pg_replication_slots + WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;} + ), + "t", + 'logical slot has synced as true on standby'); + +my $logstart = -s $primary->logfile; + +# Set timeout so that the next checkpoint will invalidate the inactive +# replication slot. +$primary->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s'; +]); +$primary->reload; + +# Wait for the logical failover slot to become inactive on the primary. Note +# that nobody has acquired that slot yet, so due to inactive timeout setting +# above it must get invalidated. +wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart); + +# Set timeout on the standby also to check the synced slots don't get +# invalidated due to timeout on the standby. +$standby1->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s'; +]); +$standby1->reload; + +# Now, sync the logical failover slot from the remote slot on the primary. +# Note that the remote slot has already been invalidated due to inactive +# timeout. Now, the standby must also see it as invalidated. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Wait for the inactive replication slot to be invalidated. +$standby1->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE slot_name = 'lsub1_sync_slot' AND + invalidation_reason = 'inactive_timeout'; +]) + or die + "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby"; + +# Synced slot mustn't get invalidated on the standby, it must sync invalidation +# from the primary. So, we must not see the slot's invalidation message in server +# log. +ok( !$standby1->log_contains( + "invalidating obsolete replication slot \"lsub1_sync_slot\"", + $standby1_logstart), + 'check that syned slot has not been invalidated on the standby'); + +# Stop standby to make the standby's replication slot on the primary inactive +$standby1->stop; + +# Wait for the standby's replication slot to become inactive +wait_for_slot_invalidation($primary, 'sb1_slot', $logstart); + +# Testcase end: Invalidate streaming standby's slot as well as logical failover +# slot on primary due to inactive timeout GUC. Also, check the logical failover +# slot synced on to the standby doesn't invalidate the slot on its own, but +# gets the invalidated state from the remote slot on the primary. +# ============================================================================= + +# ============================================================================= +# Testcase start: Invalidate logical subscriber's slot due to inactive timeout +# GUC. + +my $publisher = $primary; + +# Prepare for the next test +$publisher->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '0'; +]); +$publisher->reload; + +# Create subscriber node +my $subscriber = PostgreSQL::Test::Cluster->new('sub'); +$subscriber->init; +$subscriber->start; + +# Create tables +$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)"); +$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)"); + +# Insert some data +$subscriber->safe_psql('postgres', + "INSERT INTO test_tbl VALUES (generate_series(1, 5));"); + +# Setup logical replication +my $publisher_connstr = $publisher->connstr . ' dbname=postgres'; +$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES"); +$publisher->safe_psql( + 'postgres', qq[ + SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput'); +]); + +$subscriber->safe_psql('postgres', + "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)" +); + +$subscriber->wait_for_subscription_sync($publisher, 'sub'); + +my $result = + $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl"); + +is($result, qq(5), "check initial copy was done"); + +# Prepare for the next test +$publisher->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s'; +]); +$publisher->reload; + +$logstart = -s $publisher->logfile; + +# Stop subscriber to make the replication slot on publisher inactive +$subscriber->stop; + +# Wait for the replication slot to become inactive and then invalidated due to +# timeout. +wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart); + +# Testcase end: Invalidate logical subscriber's slot due to inactive timeout +# GUC. +# ============================================================================= + +# ============================================================================= +# Start: Helper functions used for this test file + +sub wait_for_slot_invalidation +{ + my ($node, $slot_name, $offset) = @_; + my $name = $node->name; + + # Wait for the replication slot to become inactive + $node->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND active = 'f'; + ]) + or die + "Timed out while waiting for replication slot to become inactive"; + + # Wait for the replication slot info to be updated + $node->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE inactive_since IS NOT NULL + AND slot_name = '$slot_name' AND active = 'f'; + ]) + or die + "Timed out while waiting for info of replication slot $slot_name to be updated on node $name"; + + check_for_slot_invalidation_in_server_log($node, $slot_name, $offset); + + # Wait for the inactive replication slot to be invalidated. + $node->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND + invalidation_reason = 'inactive_timeout'; + ]) + or die + "Timed out while waiting for inactive replication slot $slot_name to be invalidated on node $name"; + + # Check that the invalidated slot cannot be acquired + my ($result, $stdout, $stderr); + + ($result, $stdout, $stderr) = $primary->psql( + 'postgres', qq[ + SELECT pg_replication_slot_advance('$slot_name', '0/1'); + ]); + + ok( $stderr =~ /can no longer get changes from replication slot "$slot_name"/, + "detected error upon trying to acquire invalidated slot $slot_name on node $name" + ) + or die + "could not detect error upon trying to acquire invalidated slot $slot_name"; +} + +# Check for invalidation of slot in server log. +sub check_for_slot_invalidation_in_server_log +{ + my ($node, $slot_name, $offset) = @_; + my $invalidated = 0; + + for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++) + { + $node->safe_psql('postgres', "CHECKPOINT"); + if ($node->log_contains( + "invalidating obsolete replication slot \"$slot_name\"", + $offset)) + { + $invalidated = 1; + last; + } + usleep(100_000); + } + ok($invalidated, + "check that slot $slot_name invalidation has been logged"); +} + +# ============================================================================= +# End: Helper functions used for this test file + +done_testing(); -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-01 03:17 Bharath Rupireddy <[email protected]> parent: Amit Kapila <[email protected]> 1 sibling, 2 replies; 46+ messages in thread From: Bharath Rupireddy @ 2024-04-01 03:17 UTC (permalink / raw) To: Amit Kapila <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Fri, Mar 29, 2024 at 9:39 AM Amit Kapila <[email protected]> wrote: > > Commit message states: "why we can't just update inactive_since for > synced slots on the standby with the value received from remote slot > on the primary. This is consistent with any other slot parameter i.e. > all of them are synced from the primary." > > The inactive_since is not consistent with other slot parameters which > we copy. We don't perform anything related to those other parameters > like say two_phase phase which can change that property. However, we > do acquire the slot, advance the slot (as per recent discussion [1]), > and release it. Since these operations can impact inactive_since, it > seems to me that inactive_since is not the same as other parameters. > It can have a different value than the primary. Why would anyone want > to know the value of inactive_since from primary after the standby is > promoted? After thinking about it for a while now, it feels to me that the synced slots (slots on the standby that are being synced from the primary) can have their own inactive_sicne value. Fundamentally, inactive_sicne is set to 0 when slot is acquired and set to current time when slot is released, no matter who acquires and releases it - be it walsenders for replication, or backends for slot advance, or backends for slot sync using pg_sync_replication_slots, or backends for other slot functions, or background sync worker. Remember the earlier patch was updating inactive_since just for walsenders, but then the suggestion was to update it unconditionally - https://www.postgresql.org/message-id/CAJpy0uD64X%3D2ENmbHaRiWTKeQawr-rbGoy_GdhQQLVXzUSKTMg%40mail.g.... Whoever syncs the slot, *acutally* acquires the slot i.e. makes it theirs, syncs it from the primary, and releases it. IMO, no differentiation is to be made for synced slots. There was a suggestion on using inactive_since of the synced slot on the standby to know the inactivity of the slot on the primary. If one wants to do that, they better look at/monitor the primary slot info/logs/pg_replication_slot/whatever. I really don't see a point in having two different meanings for a single property of a replication slot - inactive_since for a regular slot tells since when this slot has become inactive, and for a synced slot since when the corresponding remote slot has become inactive. I think this will confuse users for sure. Also, if inactive_since is being changed on the primary so frequently, and none of the other parameters are changing, if we copy inactive_since to the synced slots, then standby will just be doing *sync* work (mark the slots dirty and save to disk) for updating inactive_since. I think this is unnecessary behaviour for sure. Coming to a future patch for inactive timeout based slot invalidation, we can either allow invalidation without any differentiation for synced slots or restrict invalidation to avoid more sync work. For instance, if inactive timeout is kept low on the standby, the sync worker will be doing more work as it drops and recreates a slot repeatedly if it keeps getting invalidated. Another thing is that the standby takes independent invalidation decisions for synced slots. AFAICS, invalidation due to wal_removal is the only sole reason (out of all available invalidation reasons) for a synced slot to get invalidated independently of the primary. Check https://www.postgresql.org/message-id/CAA4eK1JXBwTaDRD_%3D8t6UB1fhRNjC1C%2BgH4YdDxj_9U6djLnXw%40mail... for the suggestion on we better not differentiaing invalidation decisions for synced slots. The assumption of letting synced slots have their own inactive_since not only simplifies the code, but also looks less-confusing and more meaningful to the user. The only code that we put in on top of the committed code is to use InRecovery in place of RecoveryInProgress() in RestoreSlotFromDisk() to fix the issue raised by Shveta upthread. > Now, the other concern is that calling GetCurrentTimestamp() > could be costly when the values for the slot are not going to be > updated but if that happens we can optimize such that before acquiring > the slot we can have some minimal pre-checks to ensure whether we need > to update the slot or not. > > [1] - https://www.postgresql.org/message-id/OS0PR01MB571615D35F486080616CA841943A2%40OS0PR01MB5716.jpnprd0... A quick test with a function to measure the cost of GetCurrentTimestamp [1] on my Ubuntu dev system (an AWS EC2 c5.4xlarge instance), gives me [2]. It took 0.388 ms, 2.269 ms, 21.144 ms, 209.333 ms, 2091.174 ms, 20908.942 ms for 10K, 100K, 1million, 10million, 100million, 1billion times respectively. Costs might be different on various systems with different OS, but it gives us a rough idea. If we are too much concerned about the cost of GetCurrentTimestamp(), a possible approach is just don't set inactive_since for slots being synced on the standby. Just let the first acquisition and release after the promotion do that job. We can always call this out in the docs saying "replication slots on the streaming standbys which are being synced from the primary are not inactive in practice, so the inactive_since is always NULL for them unless the standby is promoted". [1] Datum pg_get_current_timestamp(PG_FUNCTION_ARGS) { int loops = PG_GETARG_INT32(0); TimestampTz ctime; for (int i = 0; i < loops; i++) ctime = GetCurrentTimestamp(); PG_RETURN_TIMESTAMPTZ(ctime); } [2] postgres=# \timing Timing is on. postgres=# SELECT pg_get_current_timestamp(1000000000); pg_get_current_timestamp ------------------------------- 2024-03-30 19:07:57.374797+00 (1 row) Time: 20908.942 ms (00:20.909) postgres=# SELECT pg_get_current_timestamp(100000000); pg_get_current_timestamp ------------------------------- 2024-03-30 19:08:21.038064+00 (1 row) Time: 2091.174 ms (00:02.091) postgres=# SELECT pg_get_current_timestamp(10000000); pg_get_current_timestamp ------------------------------- 2024-03-30 19:08:24.329949+00 (1 row) Time: 209.333 ms postgres=# SELECT pg_get_current_timestamp(1000000); pg_get_current_timestamp ------------------------------- 2024-03-30 19:08:26.978016+00 (1 row) Time: 21.144 ms postgres=# SELECT pg_get_current_timestamp(100000); pg_get_current_timestamp ------------------------------- 2024-03-30 19:08:29.142248+00 (1 row) Time: 2.269 ms postgres=# SELECT pg_get_current_timestamp(10000); pg_get_current_timestamp ------------------------------ 2024-03-30 19:08:31.34621+00 (1 row) Time: 0.388 ms -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-01 07:18 Bertrand Drouvot <[email protected]> parent: Bharath Rupireddy <[email protected]> 1 sibling, 0 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-04-01 07:18 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Mon, Apr 01, 2024 at 08:47:59AM +0530, Bharath Rupireddy wrote: > On Fri, Mar 29, 2024 at 9:39 AM Amit Kapila <[email protected]> wrote: > > > > Commit message states: "why we can't just update inactive_since for > > synced slots on the standby with the value received from remote slot > > on the primary. This is consistent with any other slot parameter i.e. > > all of them are synced from the primary." > > > > The inactive_since is not consistent with other slot parameters which > > we copy. We don't perform anything related to those other parameters > > like say two_phase phase which can change that property. However, we > > do acquire the slot, advance the slot (as per recent discussion [1]), > > and release it. Since these operations can impact inactive_since, it > > seems to me that inactive_since is not the same as other parameters. > > It can have a different value than the primary. Why would anyone want > > to know the value of inactive_since from primary after the standby is > > promoted? > > After thinking about it for a while now, it feels to me that the > synced slots (slots on the standby that are being synced from the > primary) can have their own inactive_sicne value. Fundamentally, > inactive_sicne is set to 0 when slot is acquired and set to current > time when slot is released, no matter who acquires and releases it - > be it walsenders for replication, or backends for slot advance, or > backends for slot sync using pg_sync_replication_slots, or backends > for other slot functions, or background sync worker. Remember the > earlier patch was updating inactive_since just for walsenders, but > then the suggestion was to update it unconditionally - > https://www.postgresql.org/message-id/CAJpy0uD64X%3D2ENmbHaRiWTKeQawr-rbGoy_GdhQQLVXzUSKTMg%40mail.g.... > Whoever syncs the slot, *acutally* acquires the slot i.e. makes it > theirs, syncs it from the primary, and releases it. IMO, no > differentiation is to be made for synced slots. > > There was a suggestion on using inactive_since of the synced slot on > the standby to know the inactivity of the slot on the primary. If one > wants to do that, they better look at/monitor the primary slot > info/logs/pg_replication_slot/whatever. Yeah but the use case was in case the primary is down for whatever reason. > I really don't see a point in > having two different meanings for a single property of a replication > slot - inactive_since for a regular slot tells since when this slot > has become inactive, and for a synced slot since when the > corresponding remote slot has become inactive. I think this will > confuse users for sure. I'm not sure as we are speaking about "synced" slots. I can also see some confusion if this value is not "synced". > Also, if inactive_since is being changed on the primary so frequently, > and none of the other parameters are changing, if we copy > inactive_since to the synced slots, then standby will just be doing > *sync* work (mark the slots dirty and save to disk) for updating > inactive_since. I think this is unnecessary behaviour for sure. Right, I think we should avoid the save slot to disk in that case (question raised by Amit in [1]). > Coming to a future patch for inactive timeout based slot invalidation, > we can either allow invalidation without any differentiation for > synced slots or restrict invalidation to avoid more sync work. For > instance, if inactive timeout is kept low on the standby, the sync > worker will be doing more work as it drops and recreates a slot > repeatedly if it keeps getting invalidated. Another thing is that the > standby takes independent invalidation decisions for synced slots. > AFAICS, invalidation due to wal_removal is the only sole reason (out > of all available invalidation reasons) for a synced slot to get > invalidated independently of the primary. Check > https://www.postgresql.org/message-id/CAA4eK1JXBwTaDRD_%3D8t6UB1fhRNjC1C%2BgH4YdDxj_9U6djLnXw%40mail... > for the suggestion on we better not differentiaing invalidation > decisions for synced slots. Yeah, I think the invalidation decision on the standby is highly linked to what inactive_since on the standby is: synced from primary or not. > The assumption of letting synced slots have their own inactive_since > not only simplifies the code, but also looks less-confusing and more > meaningful to the user. I'm not sure at all. But if the majority of us thinks it's the case then let's go that way. > > Now, the other concern is that calling GetCurrentTimestamp() > > could be costly when the values for the slot are not going to be > > updated but if that happens we can optimize such that before acquiring > > the slot we can have some minimal pre-checks to ensure whether we need > > to update the slot or not. Also maybe we could accept a bit less accuracy and use GetCurrentTransactionStopTimestamp() instead? > If we are too much concerned about the cost of GetCurrentTimestamp(), > a possible approach is just don't set inactive_since for slots being > synced on the standby. > Just let the first acquisition and release > after the promotion do that job. We can always call this out in the > docs saying "replication slots on the streaming standbys which are > being synced from the primary are not inactive in practice, so the > inactive_since is always NULL for them unless the standby is > promoted". I think that was the initial behavior that lead to Robert's remark (see [2]): " And I'm suspicious that having an exception for slots being synced is a bad idea. That makes too much of a judgement about how the user will use this field. It's usually better to just expose the data, and if the user needs helps to make sense of that data, then give them that help separately. " [1]: https://www.postgresql.org/message-id/CAA4eK1JtKieWMivbswYg5FVVB5FugCftLvQKVsxh%3Dm_8nk04vw%40mail.g... [2]: https://www.postgresql.org/message-id/CA%2BTgmob_Ta-t2ty8QrKHBGnNLrf4ZYcwhGHGFsuUoFrAEDw4sA%40mail.g... Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-01 09:59 Bertrand Drouvot <[email protected]> parent: Bharath Rupireddy <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-04-01 09:59 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Sun, Mar 31, 2024 at 10:25:46AM +0530, Bharath Rupireddy wrote: > On Thu, Mar 28, 2024 at 3:13 PM Bertrand Drouvot > <[email protected]> wrote: > > I think in this case it should always reflect the value from the primary (so > > that one can understand why it is invalidated). > > I'll come back to this as soon as we all agree on inactive_since > behavior for synced slots. Makes sense. Also if the majority of us thinks it's not needed for inactive_since to be an exact copy of the primary, then let's go that way. > > I think when it is invalidated it should always reflect the value from the > > primary (so that one can understand why it is invalidated). > > I'll come back to this as soon as we all agree on inactive_since > behavior for synced slots. Yeah. > > T4 === > > > > Also, it looks like querying pg_replication_slots() does not trigger an > > invalidation: I think it should if the slot is not invalidated yet (and matches > > the invalidation criteria). > > There's a different opinion on this, check comment #3 from > https://www.postgresql.org/message-id/CAA4eK1LLj%2BeaMN-K8oeOjfG%2BUuzTY%3DL5PXbcMJURZbFm%2B_aJSA%40.... Oh right, I can see Amit's point too. Let's put pg_replication_slots() out of the game then. > > CR6 === > > > > +static bool > > +InvalidateSlotForInactiveTimeout(ReplicationSlot *slot, bool need_locks) > > +{ > > > > InvalidatePossiblyInactiveSlot() maybe? > > I think we will lose the essence i.e. timeout from the suggested > function name, otherwise just the inactive doesn't give a clearer > meaning. I kept it that way unless anyone suggests otherwise. Right. OTOH I think that "Possibly" adds some nuance (like InvalidatePossiblyObsoleteSlot() is already doing). > Please see the attached v30 patch. 0002 is where all of the above > review comments have been addressed. Thanks! FYI, I did not look at the content yet, just replied to the above comments. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-02 03:07 Masahiko Sawada <[email protected]> parent: Bharath Rupireddy <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Masahiko Sawada @ 2024-04-02 03:07 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Mon, Apr 1, 2024 at 12:18 PM Bharath Rupireddy <[email protected]> wrote: > > On Fri, Mar 29, 2024 at 9:39 AM Amit Kapila <[email protected]> wrote: > > > > Commit message states: "why we can't just update inactive_since for > > synced slots on the standby with the value received from remote slot > > on the primary. This is consistent with any other slot parameter i.e. > > all of them are synced from the primary." > > > > The inactive_since is not consistent with other slot parameters which > > we copy. We don't perform anything related to those other parameters > > like say two_phase phase which can change that property. However, we > > do acquire the slot, advance the slot (as per recent discussion [1]), > > and release it. Since these operations can impact inactive_since, it > > seems to me that inactive_since is not the same as other parameters. > > It can have a different value than the primary. Why would anyone want > > to know the value of inactive_since from primary after the standby is > > promoted? > > After thinking about it for a while now, it feels to me that the > synced slots (slots on the standby that are being synced from the > primary) can have their own inactive_sicne value. Fundamentally, > inactive_sicne is set to 0 when slot is acquired and set to current > time when slot is released, no matter who acquires and releases it - > be it walsenders for replication, or backends for slot advance, or > backends for slot sync using pg_sync_replication_slots, or backends > for other slot functions, or background sync worker. Remember the > earlier patch was updating inactive_since just for walsenders, but > then the suggestion was to update it unconditionally - > https://www.postgresql.org/message-id/CAJpy0uD64X%3D2ENmbHaRiWTKeQawr-rbGoy_GdhQQLVXzUSKTMg%40mail.g.... > Whoever syncs the slot, *acutally* acquires the slot i.e. makes it > theirs, syncs it from the primary, and releases it. IMO, no > differentiation is to be made for synced slots. FWIW, coming to this thread late, I think that the inactive_since should not be synchronized from the primary. The wall clocks are different on the primary and the standby so having the primary's timestamp on the standby can confuse users, especially when there is a big clock drift. Also, as Amit mentioned, inactive_since seems not to be consistent with other parameters we copy. The replication_slot_inactive_timeout feature should work on the standby independent from the primary, like other slot invalidation mechanisms, and it should be based on its own local clock. > Coming to a future patch for inactive timeout based slot invalidation, > we can either allow invalidation without any differentiation for > synced slots or restrict invalidation to avoid more sync work. For > instance, if inactive timeout is kept low on the standby, the sync > worker will be doing more work as it drops and recreates a slot > repeatedly if it keeps getting invalidated. Another thing is that the > standby takes independent invalidation decisions for synced slots. > AFAICS, invalidation due to wal_removal is the only sole reason (out > of all available invalidation reasons) for a synced slot to get > invalidated independently of the primary. Check > https://www.postgresql.org/message-id/CAA4eK1JXBwTaDRD_%3D8t6UB1fhRNjC1C%2BgH4YdDxj_9U6djLnXw%40mail... > for the suggestion on we better not differentiaing invalidation > decisions for synced slots. > > The assumption of letting synced slots have their own inactive_since > not only simplifies the code, but also looks less-confusing and more > meaningful to the user. The only code that we put in on top of the > committed code is to use InRecovery in place of > RecoveryInProgress() in RestoreSlotFromDisk() to fix the issue raised > by Shveta upthread. If we want to invalidate the synced slots due to the timeout, I think we need to define what is "inactive" for synced slots. Suppose that the slotsync worker updates the local (synced) slot's inactive_since whenever releasing the slot, irrespective of the actual LSNs (or other slot parameters) having been updated. I think that this idea cannot handle a slot that is not acquired on the primary. In this case, the remote slot is inactive but the local slot is regarded as active. WAL files are piled up on the standby (and on the primary) as the slot's LSNs don't move forward. I think we want to regard such a slot as "inactive" also on the standby and invalidate it because of the timeout. > > > Now, the other concern is that calling GetCurrentTimestamp() > > could be costly when the values for the slot are not going to be > > updated but if that happens we can optimize such that before acquiring > > the slot we can have some minimal pre-checks to ensure whether we need > > to update the slot or not. If we use such pre-checks, another problem might happen; it cannot handle a case where the slot is acquired on the primary but its LSNs don't move forward. Imagine a logical replication conflict happened on the subscriber, and the logical replication enters the retry loop. In this case, the remote slot's inactive_since gets updated for every retry, but it looks inactive from the standby since the slot LSNs don't change. Therefore, only the local slot could be invalidated due to the timeout but probably we don't want to regard such a slot as "inactive". Another idea I came up with is that the slotsync worker updates the local slot's inactive_since to the local timestamp only when the remote slot might have got inactive. If the remote slot is acquired by someone, the local slot's inactive_since is also NULL. If the remote slot gets inactive, the slotsync worker sets the local timestamp to the local slot's inactive_since. Since the remote slot could be acquired and released before the slotsync worker gets the remote slot data again, if the remote slot's inactive_since > the local slot's inactive_since, the slotsync worker updates the local one. IOW, we detect whether the remote slot was acquired and released since the last synchronization, by checking the remote slot's inactive_since. This idea seems to handle these cases I mentioned unless I'm missing something, but it requires for the slotsync worker to update inactive_since in a different way than other parameters. Or a simple solution is that the slotsync worker updates inactive_since as it does for non-synced slots, and disables timeout-based slot invalidation for synced slots. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-02 06:28 Bertrand Drouvot <[email protected]> parent: Masahiko Sawada <[email protected]> 0 siblings, 2 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-04-02 06:28 UTC (permalink / raw) To: Masahiko Sawada <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Tue, Apr 02, 2024 at 12:07:54PM +0900, Masahiko Sawada wrote: > On Mon, Apr 1, 2024 at 12:18 PM Bharath Rupireddy > > FWIW, coming to this thread late, I think that the inactive_since > should not be synchronized from the primary. The wall clocks are > different on the primary and the standby so having the primary's > timestamp on the standby can confuse users, especially when there is a > big clock drift. Also, as Amit mentioned, inactive_since seems not to > be consistent with other parameters we copy. The > replication_slot_inactive_timeout feature should work on the standby > independent from the primary, like other slot invalidation mechanisms, > and it should be based on its own local clock. Thanks for sharing your thoughts! So, it looks like that most of us agree to not sync inactive_since from the primary, I'm fine with that. > If we want to invalidate the synced slots due to the timeout, I think > we need to define what is "inactive" for synced slots. > > Suppose that the slotsync worker updates the local (synced) slot's > inactive_since whenever releasing the slot, irrespective of the actual > LSNs (or other slot parameters) having been updated. I think that this > idea cannot handle a slot that is not acquired on the primary. In this > case, the remote slot is inactive but the local slot is regarded as > active. WAL files are piled up on the standby (and on the primary) as > the slot's LSNs don't move forward. I think we want to regard such a > slot as "inactive" also on the standby and invalidate it because of > the timeout. I think that makes sense to somehow link inactive_since on the standby to the actual LSNs (or other slot parameters) being updated or not. > > > Now, the other concern is that calling GetCurrentTimestamp() > > > could be costly when the values for the slot are not going to be > > > updated but if that happens we can optimize such that before acquiring > > > the slot we can have some minimal pre-checks to ensure whether we need > > > to update the slot or not. > > If we use such pre-checks, another problem might happen; it cannot > handle a case where the slot is acquired on the primary but its LSNs > don't move forward. Imagine a logical replication conflict happened on > the subscriber, and the logical replication enters the retry loop. In > this case, the remote slot's inactive_since gets updated for every > retry, but it looks inactive from the standby since the slot LSNs > don't change. Therefore, only the local slot could be invalidated due > to the timeout but probably we don't want to regard such a slot as > "inactive". > > Another idea I came up with is that the slotsync worker updates the > local slot's inactive_since to the local timestamp only when the > remote slot might have got inactive. If the remote slot is acquired by > someone, the local slot's inactive_since is also NULL. If the remote > slot gets inactive, the slotsync worker sets the local timestamp to > the local slot's inactive_since. Since the remote slot could be > acquired and released before the slotsync worker gets the remote slot > data again, if the remote slot's inactive_since > the local slot's > inactive_since, the slotsync worker updates the local one. Then I think we would need to be careful about time zone comparison. > IOW, we > detect whether the remote slot was acquired and released since the > last synchronization, by checking the remote slot's inactive_since. > This idea seems to handle these cases I mentioned unless I'm missing > something, but it requires for the slotsync worker to update > inactive_since in a different way than other parameters. > > Or a simple solution is that the slotsync worker updates > inactive_since as it does for non-synced slots, and disables > timeout-based slot invalidation for synced slots. Yeah, I think the main question to help us decide is: do we want to invalidate "inactive" synced slots locally (in addition to synchronizing the invalidation from the primary)? Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-02 07:11 Bharath Rupireddy <[email protected]> parent: Bertrand Drouvot <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: Bharath Rupireddy @ 2024-04-02 07:11 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Tue, Apr 2, 2024 at 11:58 AM Bertrand Drouvot <[email protected]> wrote: > > > Or a simple solution is that the slotsync worker updates > > inactive_since as it does for non-synced slots, and disables > > timeout-based slot invalidation for synced slots. > > Yeah, I think the main question to help us decide is: do we want to invalidate > "inactive" synced slots locally (in addition to synchronizing the invalidation > from the primary)? I think this approach looks way simpler than the other one. The other approach of linking inactive_since on the standby for synced slots to the actual LSNs (or other slot parameters) being updated or not looks more complicated, and might not go well with the end user. However, we need to be able to say why we don't invalidate synced slots due to inactive timeout unlike the wal_removed invalidation that can happen right now on the standby for synced slots. This leads us to define actually what a slot being active means. Is syncing the data from the remote slot considered as the slot being active? On the other hand, it may not sound great if we don't invalidate synced slots due to inactive timeout even though they hold resources such as WAL and XIDs. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-02 08:33 Bertrand Drouvot <[email protected]> parent: Bharath Rupireddy <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-04-02 08:33 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Tue, Apr 02, 2024 at 12:41:35PM +0530, Bharath Rupireddy wrote: > On Tue, Apr 2, 2024 at 11:58 AM Bertrand Drouvot > <[email protected]> wrote: > > > > > Or a simple solution is that the slotsync worker updates > > > inactive_since as it does for non-synced slots, and disables > > > timeout-based slot invalidation for synced slots. > > > > Yeah, I think the main question to help us decide is: do we want to invalidate > > "inactive" synced slots locally (in addition to synchronizing the invalidation > > from the primary)? > > I think this approach looks way simpler than the other one. The other > approach of linking inactive_since on the standby for synced slots to > the actual LSNs (or other slot parameters) being updated or not looks > more complicated, and might not go well with the end user. However, > we need to be able to say why we don't invalidate synced slots due to > inactive timeout unlike the wal_removed invalidation that can happen > right now on the standby for synced slots. This leads us to define > actually what a slot being active means. Is syncing the data from the > remote slot considered as the slot being active? > > On the other hand, it may not sound great if we don't invalidate > synced slots due to inactive timeout even though they hold resources > such as WAL and XIDs. Right and the "only" benefit then would be to give an idea as to when the last sync did occur on the local slot. Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-03 03:08 shveta malik <[email protected]> parent: Bertrand Drouvot <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: shveta malik @ 2024-04-03 03:08 UTC (permalink / raw) To: Bertrand Drouvot <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Tue, Apr 2, 2024 at 11:58 AM Bertrand Drouvot <[email protected]> wrote: > > Hi, > > On Tue, Apr 02, 2024 at 12:07:54PM +0900, Masahiko Sawada wrote: > > On Mon, Apr 1, 2024 at 12:18 PM Bharath Rupireddy > > > > FWIW, coming to this thread late, I think that the inactive_since > > should not be synchronized from the primary. The wall clocks are > > different on the primary and the standby so having the primary's > > timestamp on the standby can confuse users, especially when there is a > > big clock drift. Also, as Amit mentioned, inactive_since seems not to > > be consistent with other parameters we copy. The > > replication_slot_inactive_timeout feature should work on the standby > > independent from the primary, like other slot invalidation mechanisms, > > and it should be based on its own local clock. > > Thanks for sharing your thoughts! So, it looks like that most of us agree to not > sync inactive_since from the primary, I'm fine with that. +1 on not syncing slots from primary. > > If we want to invalidate the synced slots due to the timeout, I think > > we need to define what is "inactive" for synced slots. > > > > Suppose that the slotsync worker updates the local (synced) slot's > > inactive_since whenever releasing the slot, irrespective of the actual > > LSNs (or other slot parameters) having been updated. I think that this > > idea cannot handle a slot that is not acquired on the primary. In this > > case, the remote slot is inactive but the local slot is regarded as > > active. WAL files are piled up on the standby (and on the primary) as > > the slot's LSNs don't move forward. I think we want to regard such a > > slot as "inactive" also on the standby and invalidate it because of > > the timeout. > > I think that makes sense to somehow link inactive_since on the standby to > the actual LSNs (or other slot parameters) being updated or not. > > > > > Now, the other concern is that calling GetCurrentTimestamp() > > > > could be costly when the values for the slot are not going to be > > > > updated but if that happens we can optimize such that before acquiring > > > > the slot we can have some minimal pre-checks to ensure whether we need > > > > to update the slot or not. > > > > If we use such pre-checks, another problem might happen; it cannot > > handle a case where the slot is acquired on the primary but its LSNs > > don't move forward. Imagine a logical replication conflict happened on > > the subscriber, and the logical replication enters the retry loop. In > > this case, the remote slot's inactive_since gets updated for every > > retry, but it looks inactive from the standby since the slot LSNs > > don't change. Therefore, only the local slot could be invalidated due > > to the timeout but probably we don't want to regard such a slot as > > "inactive". > > > > Another idea I came up with is that the slotsync worker updates the > > local slot's inactive_since to the local timestamp only when the > > remote slot might have got inactive. If the remote slot is acquired by > > someone, the local slot's inactive_since is also NULL. If the remote > > slot gets inactive, the slotsync worker sets the local timestamp to > > the local slot's inactive_since. Since the remote slot could be > > acquired and released before the slotsync worker gets the remote slot > > data again, if the remote slot's inactive_since > the local slot's > > inactive_since, the slotsync worker updates the local one. > > Then I think we would need to be careful about time zone comparison. Yes. Also we need to consider the case when a user is relying on pg_sync_replication_slots() and has not enabled slot-sync worker. In such a case if synced slot's inactive_since is derived from inactivity of remote-slot, it might not be that frequently updated (based on when the user actually runs the SQL function) and thus may be misleading. OTOH, if inactivty_since of synced slots represents its own inactivity, then it will give correct info even for the case when the SQL function is run after a long time and slot-sync worker is disabled. > > IOW, we > > detect whether the remote slot was acquired and released since the > > last synchronization, by checking the remote slot's inactive_since. > > This idea seems to handle these cases I mentioned unless I'm missing > > something, but it requires for the slotsync worker to update > > inactive_since in a different way than other parameters. > > > > Or a simple solution is that the slotsync worker updates > > inactive_since as it does for non-synced slots, and disables > > timeout-based slot invalidation for synced slots. I like this idea better, it takes care of such a case too when the user is relying on sync-function rather than worker and does not want to get the slots invalidated in between 2 sync function calls. > Yeah, I think the main question to help us decide is: do we want to invalidate > "inactive" synced slots locally (in addition to synchronizing the invalidation > from the primary)? thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-03 05:47 Bharath Rupireddy <[email protected]> parent: shveta malik <[email protected]> 0 siblings, 2 replies; 46+ messages in thread From: Bharath Rupireddy @ 2024-04-03 05:47 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Apr 3, 2024 at 8:38 AM shveta malik <[email protected]> wrote: > > > > Or a simple solution is that the slotsync worker updates > > > inactive_since as it does for non-synced slots, and disables > > > timeout-based slot invalidation for synced slots. > > I like this idea better, it takes care of such a case too when the > user is relying on sync-function rather than worker and does not want > to get the slots invalidated in between 2 sync function calls. Please find the attached v31 patches implementing the above idea: - synced slots get their on inactive_since just like any other slot - synced slots don't get invalidated due to inactive timeout because such slots not considered active at all as they don't perform logical decoding (of course, they will perform in fast_forward mode to fix the other data loss issue, but they don't generate changes for them to be called as *active* slots) - synced slots inactive_since is set to current timestamp after the standby gets promoted to help inactive_since interpret correctly just like any other slot. -- Bharath Rupireddy PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com Attachments: [application/x-patch] v31-0001-Allow-synced-slots-to-have-their-own-inactive_si.patch (12.1K, ../../CALj2ACVfDBcbybgyga7ia_KiQPAfEi_3=VDOGf2VU8RkJZSQpw@mail.gmail.com/2-v31-0001-Allow-synced-slots-to-have-their-own-inactive_si.patch) download | inline diff: From fd2cb48726dd4e1932f8809dfb36e0fe9f922226 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Wed, 3 Apr 2024 05:03:22 +0000 Subject: [PATCH v31 1/2] Allow synced slots to have their own inactive_since. --- doc/src/sgml/system-views.sgml | 7 +++ src/backend/replication/logical/slotsync.c | 44 +++++++++++++++++ src/backend/replication/slot.c | 23 +++------ src/test/perl/PostgreSQL/Test/Cluster.pm | 34 +++++++++++++ src/test/recovery/t/019_replslot_limit.pl | 26 +--------- .../t/040_standby_failover_slots_sync.pl | 49 +++++++++++++++++++ 6 files changed, 144 insertions(+), 39 deletions(-) diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index 3c8dca8ca3..b64274a1fb 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2530,6 +2530,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx <para> The time since the slot has become inactive. <literal>NULL</literal> if the slot is currently being used. + Note that for slots on the standby that are being synced from a + primary server (whose <structfield>synced</structfield> field is + <literal>true</literal>), the + <structfield>inactive_since</structfield> value will get updated + after every synchronization (see + <xref linkend="logicaldecoding-replication-slots-synchronization"/>) + from the corresponding remote slot on the primary. </para></entry> </row> diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 30480960c5..4050bd40f8 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -140,6 +140,7 @@ typedef struct RemoteSlot } RemoteSlot; static void slotsync_failure_callback(int code, Datum arg); +static void update_synced_slots_inactive_since(void); /* * If necessary, update the local synced slot's metadata based on the data @@ -1296,6 +1297,46 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len) Assert(false); } +/* + * Update the inactive_since property for synced slots. + */ +static void +update_synced_slots_inactive_since(void) +{ + TimestampTz now = 0; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + + for (int i = 0; i < max_replication_slots; i++) + { + ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i]; + + /* Check if it is a synchronized slot */ + if (s->in_use && s->data.synced) + { + Assert(SlotIsLogical(s)); + + /* + * We get the current time beforehand and only once to avoid + * system calls overhead while holding the lock. + */ + if (now == 0) + now = GetCurrentTimestamp(); + + /* + * Set the time since the slot has become inactive after shutting + * down slot sync machinery. This helps correctly interpret the + * time if the standby gets promoted without a restart. + */ + SpinLockAcquire(&s->mutex); + s->inactive_since = now; + SpinLockRelease(&s->mutex); + } + } + + LWLockRelease(ReplicationSlotControlLock); +} + /* * Shut down the slot sync worker. */ @@ -1309,6 +1350,7 @@ ShutDownSlotSync(void) if (SlotSyncCtx->pid == InvalidPid) { SpinLockRelease(&SlotSyncCtx->mutex); + update_synced_slots_inactive_since(); return; } SpinLockRelease(&SlotSyncCtx->mutex); @@ -1341,6 +1383,8 @@ ShutDownSlotSync(void) } SpinLockRelease(&SlotSyncCtx->mutex); + + update_synced_slots_inactive_since(); } /* diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index d778c0b921..5549ca9640 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -42,6 +42,7 @@ #include "access/transam.h" #include "access/xlog_internal.h" #include "access/xlogrecovery.h" +#include "access/xlogutils.h" #include "common/file_utils.h" #include "common/string.h" #include "miscadmin.h" @@ -690,13 +691,10 @@ ReplicationSlotRelease(void) } /* - * Set the last inactive time after marking the slot inactive. We don't - * set it for the slots currently being synced from the primary to the - * standby because such slots are typically inactive as decoding is not - * allowed on those. + * Set the time since the slot has become inactive. We get the current + * time beforehand to avoid system call overhead while holding the lock. */ - if (!(RecoveryInProgress() && slot->data.synced)) - now = GetCurrentTimestamp(); + now = GetCurrentTimestamp(); if (slot->data.persistency == RS_PERSISTENT) { @@ -2369,16 +2367,11 @@ RestoreSlotFromDisk(const char *name) slot->active_pid = 0; /* - * We set the last inactive time after loading the slot from the disk - * into memory. Whoever acquires the slot i.e. makes the slot active - * will reset it. We don't set it for the slots currently being synced - * from the primary to the standby because such slots are typically - * inactive as decoding is not allowed on those. + * Set the time since the slot has become inactive after loading the + * slot from the disk into memory. Whoever acquires the slot i.e. + * makes the slot active will reset it. */ - if (!(RecoveryInProgress() && slot->data.synced)) - slot->inactive_since = GetCurrentTimestamp(); - else - slot->inactive_since = 0; + slot->inactive_since = GetCurrentTimestamp(); restored = true; break; diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm index b08296605c..ddfc3236f3 100644 --- a/src/test/perl/PostgreSQL/Test/Cluster.pm +++ b/src/test/perl/PostgreSQL/Test/Cluster.pm @@ -3276,6 +3276,40 @@ sub create_logical_slot_on_standby =pod +=item $node->get_slot_inactive_since_value(self, slot_name, reference_time) + +Get inactive_since column value for a given replication slot validating it +against optional reference time. + +=cut + +sub get_slot_inactive_since_value +{ + my ($self, $slot_name, $reference_time) = @_; + my $name = $self->name; + + my $inactive_since = $self->safe_psql('postgres', + qq(SELECT inactive_since FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) + ); + + # Check that the captured time is sane + if (defined $reference_time) + { + is($self->safe_psql('postgres', + qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND + '$inactive_since'::timestamptz >= '$reference_time'::timestamptz;] + ), + 't', + "last inactive time for slot $slot_name is valid on node $name") + or die "could not validate captured inactive_since for slot $slot_name"; + } + + return $inactive_since; +} + +=pod + =item $node->advance_wal(num) Advance WAL of node by given number of segments. diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl index 3b9a306a8b..c8e5e5054d 100644 --- a/src/test/recovery/t/019_replslot_limit.pl +++ b/src/test/recovery/t/019_replslot_limit.pl @@ -443,7 +443,7 @@ $primary4->safe_psql( # Get inactive_since value after the slot's creation. Note that the slot is # still inactive till it's used by the standby below. my $inactive_since = - capture_and_validate_slot_inactive_since($primary4, $sb4_slot, $slot_creation_time); + $primary4->get_slot_inactive_since_value($sb4_slot, $slot_creation_time); $standby4->start; @@ -502,7 +502,7 @@ $publisher4->safe_psql('postgres', # Get inactive_since value after the slot's creation. Note that the slot is # still inactive till it's used by the subscriber below. $inactive_since = - capture_and_validate_slot_inactive_since($publisher4, $lsub4_slot, $slot_creation_time); + $publisher4->get_slot_inactive_since_value($lsub4_slot, $slot_creation_time); $subscriber4->start; $subscriber4->safe_psql('postgres', @@ -540,26 +540,4 @@ is( $publisher4->safe_psql( $publisher4->stop; $subscriber4->stop; -# Capture and validate inactive_since of a given slot. -sub capture_and_validate_slot_inactive_since -{ - my ($node, $slot_name, $slot_creation_time) = @_; - - my $inactive_since = $node->safe_psql('postgres', - qq(SELECT inactive_since FROM pg_replication_slots - WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;) - ); - - # Check that the captured time is sane - is( $node->safe_psql( - 'postgres', - qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND - '$inactive_since'::timestamptz >= '$slot_creation_time'::timestamptz;] - ), - 't', - "last inactive time for an active slot $slot_name is sane"); - - return $inactive_since; -} - done_testing(); diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl index f47bfd78eb..50be94e629 100644 --- a/src/test/recovery/t/040_standby_failover_slots_sync.pl +++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl @@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1'); $subscriber1->init; $subscriber1->start; +# Capture the time before the logical failover slot is created on the +# primary. We later call this publisher as primary anyway. +my $slot_creation_time_on_primary = $publisher->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + # Create a slot on the publisher with failover disabled $publisher->safe_psql('postgres', "SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);" @@ -174,6 +181,10 @@ $primary->poll_query_until( "SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'", 1); +# Capture the inactive_since of the slot from the primary +my $inactive_since_on_primary = + $primary->get_slot_inactive_since_value('lsub1_slot', $slot_creation_time_on_primary); + # Wait for the standby to catch up so that the standby is not lagging behind # the subscriber. $primary->wait_for_replay_catchup($standby1); @@ -181,6 +192,11 @@ $primary->wait_for_replay_catchup($standby1); # Synchronize the primary server slots to the standby. $standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); +my $slot_sync_time = $standby1->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + # Confirm that the logical failover slots are created on the standby and are # flagged as 'synced' is( $standby1->safe_psql( @@ -190,6 +206,19 @@ is( $standby1->safe_psql( "t", 'logical slots have synced as true on standby'); +# Capture the inactive_since of the synced slot on the standby +my $inactive_since_on_standby = + $standby1->get_slot_inactive_since_value('lsub1_slot', $slot_creation_time_on_primary); + +# Synced slot on the standby must get its own inactive_since. +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_primary'::timestamptz < '$inactive_since_on_standby'::timestamptz AND + '$inactive_since_on_standby'::timestamptz < '$slot_sync_time'::timestamptz;" + ), + "t", + 'synchronized slot has got its own inactive_since'); + ################################################## # Test that the synchronized slot will be dropped if the corresponding remote # slot on the primary server has been dropped. @@ -750,8 +779,28 @@ $primary->reload; $standby1->start; $primary->wait_for_replay_catchup($standby1); +# Capture the time before the standby is promoted +my $promotion_time_on_primary = $standby1->safe_psql( + 'postgres', qq[ + SELECT current_timestamp; +]); + $standby1->promote; +# Capture the inactive_since of the synced slot after the promotion. +# Expectation here is that the slot gets its own inactive_since as part of the +# promotion. We do this check before the slot is enabled on the new primary +# below, otherwise the slot gets active setting inactive_since to NULL. +my $inactive_since_on_new_primary = + $standby1->get_slot_inactive_since_value('lsub1_slot', $promotion_time_on_primary); + +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_new_primary'::timestamptz > '$inactive_since_on_primary'::timestamptz" + ), + "t", + 'synchronized slot has got its own inactive_since on the new primary after promotion'); + # Update subscription with the new primary's connection info my $standby1_conninfo = $standby1->connstr . ' dbname=postgres'; $subscriber1->safe_psql('postgres', -- 2.34.1 [application/x-patch] v31-0002-Add-inactive_timeout-based-replication-slot-inva.patch (32.1K, ../../CALj2ACVfDBcbybgyga7ia_KiQPAfEi_3=VDOGf2VU8RkJZSQpw@mail.gmail.com/3-v31-0002-Add-inactive_timeout-based-replication-slot-inva.patch) download | inline diff: From 382a63bd4c48318ec53bdbe292b3ab1ecbaeb341 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <[email protected]> Date: Wed, 3 Apr 2024 05:32:14 +0000 Subject: [PATCH v31 2/2] Add inactive_timeout based replication slot invalidation. Till now, postgres has the ability to invalidate inactive replication slots based on the amount of WAL (set via max_slot_wal_keep_size GUC) that will be needed for the slots in case they become active. However, choosing a default value for max_slot_wal_keep_size is tricky. Because the amount of WAL a customer generates, and their allocated storage will vary greatly in production, making it difficult to pin down a one-size-fits-all value. It is often easy for developers to set a timeout of say 1 or 2 or 3 days at slot level, after which the inactive slots get dropped. To achieve the above, postgres introduces a GUC allowing users set inactive timeout and then a slot stays inactive for this much amount of time it invalidates the slot. The invalidation check happens at various locations to help being as latest as possible, these locations include the following: - Whenever the slot is acquired and the slot acquisition errors out if invalidated. - During checkpoint Note that this new invalidation mechanism won't kick-in for the slots that are currently being synced from the primary to the standby. Author: Bharath Rupireddy Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com --- doc/src/sgml/config.sgml | 33 +++ doc/src/sgml/system-views.sgml | 7 + .../replication/logical/logicalfuncs.c | 2 +- src/backend/replication/logical/slotsync.c | 4 +- src/backend/replication/slot.c | 203 ++++++++++++- src/backend/replication/slotfuncs.c | 2 +- src/backend/replication/walsender.c | 4 +- src/backend/utils/adt/pg_upgrade_support.c | 2 +- src/backend/utils/misc/guc_tables.c | 12 + src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/replication/slot.h | 8 +- src/test/recovery/meson.build | 1 + src/test/recovery/t/044_invalidate_slots.pl | 279 ++++++++++++++++++ 13 files changed, 534 insertions(+), 24 deletions(-) create mode 100644 src/test/recovery/t/044_invalidate_slots.pl diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 624518e0b0..0a5ed00d8d 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4547,6 +4547,39 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows </listitem> </varlistentry> + <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout"> + <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>) + <indexterm> + <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary> + </indexterm> + </term> + <listitem> + <para> + Invalidates replication slots that are inactive for longer the + specified amount of time. If this value is specified without units, + it is taken as seconds. A value of zero (which is default) disables + the timeout mechanism. This parameter can only be set in the + <filename>postgresql.conf</filename> file or on the server + command line. + </para> + + <para> + The timeout is measured from the time since the slot has become + inactive (known from its + <structfield>inactive_since</structfield> value) until it gets + used (i.e., its <structfield>active</structfield> is set to true). + </para> + + <para> + Note that this inactive timeout invalidation mechanism is not + applicable for slots on the standby that are being synced from a + primary server (whose <structfield>synced</structfield> field is + <literal>true</literal>) as such synced slots don't actually perform + logical decoding. + </para> + </listitem> + </varlistentry> + <varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp"> <term><varname>track_commit_timestamp</varname> (<type>boolean</type>) <indexterm> diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index b64274a1fb..3d33afa796 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -2580,6 +2580,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx perform logical decoding. It is set only for logical slots. </para> </listitem> + <listitem> + <para> + <literal>inactive_timeout</literal> means that the slot has been + inactive for the duration specified by + <xref linkend="guc-replication-slot-inactive-timeout"/> parameter. + </para> + </listitem> </itemizedlist> </para></entry> </row> diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c index b4dd5cce75..56fc1a45a9 100644 --- a/src/backend/replication/logical/logicalfuncs.c +++ b/src/backend/replication/logical/logicalfuncs.c @@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin else end_of_wal = GetXLogReplayRecPtr(NULL); - ReplicationSlotAcquire(NameStr(*name), true); + ReplicationSlotAcquire(NameStr(*name), true, true); PG_TRY(); { diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c index 4050bd40f8..08de929970 100644 --- a/src/backend/replication/logical/slotsync.c +++ b/src/backend/replication/logical/slotsync.c @@ -317,7 +317,7 @@ drop_local_obsolete_slots(List *remote_slot_list) if (synced_slot) { - ReplicationSlotAcquire(NameStr(local_slot->data.name), true); + ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false); ReplicationSlotDropAcquired(); } @@ -527,7 +527,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid) * InvalidatePossiblyObsoleteSlot() where it invalidates slot directly * if the slot is not acquired by other processes. */ - ReplicationSlotAcquire(remote_slot->name, true); + ReplicationSlotAcquire(remote_slot->name, true, false); Assert(slot == MyReplicationSlot); diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c index 5549ca9640..e583d0ee82 100644 --- a/src/backend/replication/slot.c +++ b/src/backend/replication/slot.c @@ -108,10 +108,11 @@ const char *const SlotInvalidationCauses[] = { [RS_INVAL_WAL_REMOVED] = "wal_removed", [RS_INVAL_HORIZON] = "rows_removed", [RS_INVAL_WAL_LEVEL] = "wal_level_insufficient", + [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout", }; /* Maximum number of invalidation causes */ -#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL +#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1), "array length mismatch"); @@ -141,6 +142,7 @@ ReplicationSlot *MyReplicationSlot = NULL; /* GUC variables */ int max_replication_slots = 10; /* the maximum number of replication * slots */ +int replication_slot_inactive_timeout = 0; /* * This GUC lists streaming replication standby server slot names that @@ -159,6 +161,7 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr; static void ReplicationSlotShmemExit(int code, Datum arg); static void ReplicationSlotDropPtr(ReplicationSlot *slot); +static bool InvalidatePossiblyInactiveSlot(ReplicationSlot *slot); /* internal persistency functions */ static void RestoreSlotFromDisk(const char *name); @@ -536,9 +539,14 @@ ReplicationSlotName(int index, Name name) * * An error is raised if nowait is true and the slot is currently in use. If * nowait is false, we sleep until the slot is released by the owning process. + * + * If check_for_timeout_invalidation is true, the slot is checked for + * invalidation based on replication_slot_inactive_timeout GUC, and an error is + * raised after making the slot ours. */ void -ReplicationSlotAcquire(const char *name, bool nowait) +ReplicationSlotAcquire(const char *name, bool nowait, + bool check_for_timeout_invalidation) { ReplicationSlot *s; int active_pid; @@ -616,6 +624,34 @@ retry: /* We made this slot active, so it's ours now. */ MyReplicationSlot = s; + /* + * Check if the given slot can be invalidated based on its inactive + * timeout. If yes, persist the invalidated state to disk and then error + * out. We do this only after making the slot ours to avoid anyone else + * acquiring it while we check for its invalidation. + */ + if (check_for_timeout_invalidation) + { + /* The slot is ours by now */ + Assert(s->active_pid == MyProcPid); + + if (InvalidateInactiveReplicationSlot(s, true)) + { + /* + * If the slot has been invalidated, recalculate the resource + * limits. + */ + ReplicationSlotsComputeRequiredXmin(false); + ReplicationSlotsComputeRequiredLSN(); + + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("can no longer get changes from replication slot \"%s\"", + NameStr(MyReplicationSlot->data.name)), + errdetail("This slot has been invalidated because it was inactive for more than the time specified by replication_slot_inactive_timeout parameter."))); + } + } + /* * The call to pgstat_acquire_replslot() protects against stats for a * different slot, from before a restart or such, being present during @@ -782,7 +818,7 @@ ReplicationSlotDrop(const char *name, bool nowait) { Assert(MyReplicationSlot == NULL); - ReplicationSlotAcquire(name, nowait); + ReplicationSlotAcquire(name, nowait, false); /* * Do not allow users to drop the slots which are currently being synced @@ -805,7 +841,7 @@ ReplicationSlotAlter(const char *name, bool failover) { Assert(MyReplicationSlot == NULL); - ReplicationSlotAcquire(name, false); + ReplicationSlotAcquire(name, false, true); if (SlotIsPhysical(MyReplicationSlot)) ereport(ERROR, @@ -981,6 +1017,20 @@ ReplicationSlotDropPtr(ReplicationSlot *slot) LWLockRelease(ReplicationSlotAllocationLock); } +/* + * Helper for ReplicationSlotSave + */ +static inline void +SaveGivenReplicationSlot(ReplicationSlot *slot, int elevel) +{ + char path[MAXPGPATH]; + + Assert(slot != NULL); + + sprintf(path, "pg_replslot/%s", NameStr(slot->data.name)); + SaveSlotToPath(slot, path, elevel); +} + /* * Serialize the currently acquired slot's state from memory to disk, thereby * guaranteeing the current state will survive a crash. @@ -988,12 +1038,21 @@ ReplicationSlotDropPtr(ReplicationSlot *slot) void ReplicationSlotSave(void) { - char path[MAXPGPATH]; + SaveGivenReplicationSlot(MyReplicationSlot, ERROR); +} - Assert(MyReplicationSlot != NULL); +/* + * Helper for ReplicationSlotMarkDirty + */ +static inline void +MarkGivenReplicationSlotDirty(ReplicationSlot *slot) +{ + Assert(slot != NULL); - sprintf(path, "pg_replslot/%s", NameStr(MyReplicationSlot->data.name)); - SaveSlotToPath(MyReplicationSlot, path, ERROR); + SpinLockAcquire(&slot->mutex); + slot->just_dirtied = true; + slot->dirty = true; + SpinLockRelease(&slot->mutex); } /* @@ -1006,14 +1065,7 @@ ReplicationSlotSave(void) void ReplicationSlotMarkDirty(void) { - ReplicationSlot *slot = MyReplicationSlot; - - Assert(MyReplicationSlot != NULL); - - SpinLockAcquire(&slot->mutex); - MyReplicationSlot->just_dirtied = true; - MyReplicationSlot->dirty = true; - SpinLockRelease(&slot->mutex); + MarkGivenReplicationSlotDirty(MyReplicationSlot); } /* @@ -1507,6 +1559,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause, case RS_INVAL_WAL_LEVEL: appendStringInfoString(&err_detail, _("Logical decoding on standby requires wal_level >= logical on the primary server.")); break; + case RS_INVAL_INACTIVE_TIMEOUT: + appendStringInfoString(&err_detail, _("The slot has been inactive for more than the time specified by replication_slot_inactive_timeout parameter.")); + break; case RS_INVAL_NONE: pg_unreachable(); } @@ -1551,6 +1606,13 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr; ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE; + /* + * This function isn't expected to be called for inactive timeout based + * invalidation. A separate function InvalidateInactiveReplicationSlot is + * to be used for that. + */ + Assert(cause != RS_INVAL_INACTIVE_TIMEOUT); + for (;;) { XLogRecPtr restart_lsn; @@ -1620,6 +1682,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, if (SlotIsLogical(s)) invalidation_cause = cause; break; + case RS_INVAL_INACTIVE_TIMEOUT: + /* not reachable */ + Assert(false); + break; case RS_INVAL_NONE: pg_unreachable(); } @@ -1773,6 +1839,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause, * - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given * db; dboid may be InvalidOid for shared relations * - RS_INVAL_WAL_LEVEL: is logical + * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout occurs * * NB - this runs as part of checkpoint, so avoid raising errors if possible. */ @@ -1788,6 +1855,13 @@ InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause cause, Assert(cause != RS_INVAL_WAL_REMOVED || oldestSegno > 0); Assert(cause != RS_INVAL_NONE); + /* + * This function isn't expected to be called for inactive timeout based + * invalidation. A separate function InvalidateInactiveReplicationSlot is + * to be used for that. + */ + Assert(cause != RS_INVAL_INACTIVE_TIMEOUT); + if (max_replication_slots == 0) return invalidated; @@ -1824,6 +1898,88 @@ restart: return invalidated; } +/* + * Invalidate given slot based on replication_slot_inactive_timeout GUC. + * + * Returns true if the slot has got invalidated. + * + * NB - this function also runs as part of checkpoint, so avoid raising errors + * if possible. + */ +bool +InvalidateInactiveReplicationSlot(ReplicationSlot *slot, bool persist_state) +{ + if (!InvalidatePossiblyInactiveSlot(slot)) + return false; + + /* Make sure the invalidated state persists across server restart */ + MarkGivenReplicationSlotDirty(slot); + + if (persist_state) + SaveGivenReplicationSlot(slot, ERROR); + + ReportSlotInvalidation(RS_INVAL_INACTIVE_TIMEOUT, false, 0, + slot->data.name, InvalidXLogRecPtr, + InvalidXLogRecPtr, InvalidTransactionId); + + return true; +} + +/* + * Helper for InvalidateInactiveReplicationSlot + */ +static bool +InvalidatePossiblyInactiveSlot(ReplicationSlot *slot) +{ + ReplicationSlotInvalidationCause inavidation_cause = RS_INVAL_NONE; + + /* + * Note that we don't invalidate slot on the standby that's currently + * being synced from the primary, because such slots are typically + * considered not active as they don't actually perform logical decoding. + */ + if (RecoveryInProgress() && slot->data.synced) + return false; + + if (replication_slot_inactive_timeout == 0) + return false; + else if (slot->inactive_since > 0) + { + TimestampTz now; + + /* + * Do not invalidate the slots which are currently being synced from + * the primary to the standby. + */ + if (RecoveryInProgress() && slot->data.synced) + return false; + + LWLockAcquire(ReplicationSlotControlLock, LW_SHARED); + SpinLockAcquire(&slot->mutex); + + /* + * Check if the slot needs to be invalidated due to + * replication_slot_inactive_timeout GUC. We do this with the spinlock + * held to avoid race conditions -- for example the inactive_since + * could change, or the slot could be dropped. + */ + now = GetCurrentTimestamp(); + if (TimestampDifferenceExceeds(slot->inactive_since, now, + replication_slot_inactive_timeout * 1000)) + { + inavidation_cause = RS_INVAL_INACTIVE_TIMEOUT; + slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT; + } + + SpinLockRelease(&slot->mutex); + LWLockRelease(ReplicationSlotControlLock); + + return (inavidation_cause == RS_INVAL_INACTIVE_TIMEOUT); + } + + return false; +} + /* * Flush all replication slots to disk. * @@ -1836,6 +1992,7 @@ void CheckPointReplicationSlots(bool is_shutdown) { int i; + bool invalidated = false; elog(DEBUG1, "performing replication slot checkpoint"); @@ -1859,6 +2016,13 @@ CheckPointReplicationSlots(bool is_shutdown) /* save the slot to disk, locking is handled in SaveSlotToPath() */ sprintf(path, "pg_replslot/%s", NameStr(s->data.name)); + /* + * Here's an opportunity to invalidate inactive replication slots + * based on timeout, so let's do it. + */ + if (InvalidateInactiveReplicationSlot(s, false)) + invalidated = true; + /* * Slot's data is not flushed each time the confirmed_flush LSN is * updated as that could lead to frequent writes. However, we decide @@ -1885,6 +2049,13 @@ CheckPointReplicationSlots(bool is_shutdown) SaveSlotToPath(s, path, LOG); } LWLockRelease(ReplicationSlotAllocationLock); + + /* If the slot has been invalidated, recalculate the resource limits */ + if (invalidated) + { + ReplicationSlotsComputeRequiredXmin(false); + ReplicationSlotsComputeRequiredLSN(); + } } /* diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index da57177c25..677c0bf0a2 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -651,7 +651,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS) moveto = Min(moveto, GetXLogReplayRecPtr(NULL)); /* Acquire the slot so we "own" it */ - ReplicationSlotAcquire(NameStr(*slotname), true); + ReplicationSlotAcquire(NameStr(*slotname), true, true); /* A slot whose restart_lsn has never been reserved cannot be advanced */ if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn)) diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c index bc40c454de..96eeb8b7d2 100644 --- a/src/backend/replication/walsender.c +++ b/src/backend/replication/walsender.c @@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd) if (cmd->slotname) { - ReplicationSlotAcquire(cmd->slotname, true); + ReplicationSlotAcquire(cmd->slotname, true, true); if (SlotIsLogical(MyReplicationSlot)) ereport(ERROR, (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), @@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd) Assert(!MyReplicationSlot); - ReplicationSlotAcquire(cmd->slotname, true); + ReplicationSlotAcquire(cmd->slotname, true, true); /* * Force a disconnect, so that the decoding code doesn't need to care diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c index c54b08fe18..82956d58d3 100644 --- a/src/backend/utils/adt/pg_upgrade_support.c +++ b/src/backend/utils/adt/pg_upgrade_support.c @@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS) slot_name = PG_GETARG_NAME(0); /* Acquire the given slot */ - ReplicationSlotAcquire(NameStr(*slot_name), true); + ReplicationSlotAcquire(NameStr(*slot_name), true, false); Assert(SlotIsLogical(MyReplicationSlot)); diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index c12784cbec..4149ff1ffe 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -2971,6 +2971,18 @@ struct config_int ConfigureNamesInt[] = NULL, NULL, NULL }, + { + {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING, + gettext_noop("Sets the amount of time to wait before invalidating an " + "inactive replication slot."), + NULL, + GUC_UNIT_S + }, + &replication_slot_inactive_timeout, + 0, 0, INT_MAX, + NULL, NULL, NULL + }, + { {"commit_delay", PGC_SUSET, WAL_SETTINGS, gettext_noop("Sets the delay in microseconds between transaction commit and " diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index baecde2841..2e1ad2eaca 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -335,6 +335,7 @@ #wal_sender_timeout = 60s # in milliseconds; 0 disables #track_commit_timestamp = off # collect timestamp of transaction commit # (change requires restart) +#replication_slot_inactive_timeout = 0 # in seconds; 0 disables # - Primary Server - diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h index 7b937d1a0c..f0ac324ce9 100644 --- a/src/include/replication/slot.h +++ b/src/include/replication/slot.h @@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause RS_INVAL_HORIZON, /* wal_level insufficient for slot */ RS_INVAL_WAL_LEVEL, + /* inactive slot timeout has occurred */ + RS_INVAL_INACTIVE_TIMEOUT, } ReplicationSlotInvalidationCause; extern PGDLLIMPORT const char *const SlotInvalidationCauses[]; @@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot; /* GUCs */ extern PGDLLIMPORT int max_replication_slots; extern PGDLLIMPORT char *standby_slot_names; +extern PGDLLIMPORT int replication_slot_inactive_timeout; /* shmem initialization functions */ extern Size ReplicationSlotsShmemSize(void); @@ -245,7 +248,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait); extern void ReplicationSlotDropAcquired(void); extern void ReplicationSlotAlter(const char *name, bool failover); -extern void ReplicationSlotAcquire(const char *name, bool nowait); +extern void ReplicationSlotAcquire(const char *name, bool nowait, + bool check_for_timeout_invalidation); extern void ReplicationSlotRelease(void); extern void ReplicationSlotCleanup(void); extern void ReplicationSlotSave(void); @@ -264,6 +268,8 @@ extern bool InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause XLogSegNo oldestSegno, Oid dboid, TransactionId snapshotConflictHorizon); +extern bool InvalidateInactiveReplicationSlot(ReplicationSlot *slot, + bool persist_state); extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_lock); extern int ReplicationSlotIndex(ReplicationSlot *slot); extern bool ReplicationSlotName(int index, Name name); diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build index 712924c2fa..0437ab5c46 100644 --- a/src/test/recovery/meson.build +++ b/src/test/recovery/meson.build @@ -52,6 +52,7 @@ tests += { 't/041_checkpoint_at_promote.pl', 't/042_low_level_backup.pl', 't/043_wal_replay_wait.pl', + 't/044_invalidate_slots.pl', ], }, } diff --git a/src/test/recovery/t/044_invalidate_slots.pl b/src/test/recovery/t/044_invalidate_slots.pl new file mode 100644 index 0000000000..0f49d16ff7 --- /dev/null +++ b/src/test/recovery/t/044_invalidate_slots.pl @@ -0,0 +1,279 @@ +# Copyright (c) 2024, PostgreSQL Global Development Group + +# Test for replication slots invalidation +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Utils; +use PostgreSQL::Test::Cluster; +use Test::More; +use Time::HiRes qw(usleep); + +# ============================================================================= +# Testcase start: Invalidate streaming standby's slot as well as logical +# failover slot on primary due to inactive timeout GUC. Also, check the logical +# failover slot synced on to the standby doesn't invalidate the slot on its own, +# but gets the invalidated state from the remote slot on the primary. + +# Initialize primary node +my $primary = PostgreSQL::Test::Cluster->new('primary'); +$primary->init(allows_streaming => 'logical'); + +# Avoid checkpoint during the test, otherwise, the test can get unpredictable +$primary->append_conf( + 'postgresql.conf', q{ +checkpoint_timeout = 1h +autovacuum = off +}); +$primary->start; + +# Take backup +my $backup_name = 'my_backup'; +$primary->backup($backup_name); + +# Create a standby linking to the primary using the replication slot +my $standby1 = PostgreSQL::Test::Cluster->new('standby1'); +$standby1->init_from_backup($primary, $backup_name, has_streaming => 1); + +my $connstr_1 = $primary->connstr; +$standby1->append_conf( + 'postgresql.conf', qq( +hot_standby_feedback = on +primary_slot_name = 'sb1_slot' +primary_conninfo = '$connstr_1 dbname=postgres' +)); + +# Create sync slot on the primary +$primary->psql('postgres', + q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);} +); + +$primary->safe_psql( + 'postgres', qq[ + SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot'); +]); + +$standby1->start; + +my $standby1_logstart = -s $standby1->logfile; + +# Wait until standby has replayed enough data +$primary->wait_for_catchup($standby1); + +# Synchronize the primary server slots to the standby. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Confirm that the logical failover slot is created on the standby and is +# flagged as 'synced'. +is( $standby1->safe_psql( + 'postgres', + q{SELECT count(*) = 1 FROM pg_replication_slots + WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;} + ), + "t", + 'logical slot has synced as true on standby'); + +my $logstart = -s $primary->logfile; + +# Set timeout so that the next checkpoint will invalidate the inactive +# replication slot. +$primary->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s'; +]); +$primary->reload; + +# Wait for the logical failover slot to become inactive on the primary. Note +# that nobody has acquired that slot yet, so due to inactive timeout setting +# above it must get invalidated. +wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart); + +# Set timeout on the standby also to check the synced slots don't get +# invalidated due to timeout on the standby. +$standby1->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s'; +]); +$standby1->reload; + +# Now, sync the logical failover slot from the remote slot on the primary. +# Note that the remote slot has already been invalidated due to inactive +# timeout. Now, the standby must also see it as invalidated. +$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();"); + +# Wait for the inactive replication slot to be invalidated. +$standby1->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE slot_name = 'lsub1_sync_slot' AND + invalidation_reason = 'inactive_timeout'; +]) + or die + "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby"; + +# Synced slot mustn't get invalidated on the standby, it must sync invalidation +# from the primary. So, we must not see the slot's invalidation message in server +# log. +ok( !$standby1->log_contains( + "invalidating obsolete replication slot \"lsub1_sync_slot\"", + $standby1_logstart), + 'check that syned slot has not been invalidated on the standby'); + +# Stop standby to make the standby's replication slot on the primary inactive +$standby1->stop; + +# Wait for the standby's replication slot to become inactive +wait_for_slot_invalidation($primary, 'sb1_slot', $logstart); + +# Testcase end: Invalidate streaming standby's slot as well as logical failover +# slot on primary due to inactive timeout GUC. Also, check the logical failover +# slot synced on to the standby doesn't invalidate the slot on its own, but +# gets the invalidated state from the remote slot on the primary. +# ============================================================================= + +# ============================================================================= +# Testcase start: Invalidate logical subscriber's slot due to inactive timeout +# GUC. + +my $publisher = $primary; + +# Prepare for the next test +$publisher->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '0'; +]); +$publisher->reload; + +# Create subscriber node +my $subscriber = PostgreSQL::Test::Cluster->new('sub'); +$subscriber->init; +$subscriber->start; + +# Create tables +$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)"); +$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)"); + +# Insert some data +$subscriber->safe_psql('postgres', + "INSERT INTO test_tbl VALUES (generate_series(1, 5));"); + +# Setup logical replication +my $publisher_connstr = $publisher->connstr . ' dbname=postgres'; +$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES"); +$publisher->safe_psql( + 'postgres', qq[ + SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput'); +]); + +$subscriber->safe_psql('postgres', + "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)" +); + +$subscriber->wait_for_subscription_sync($publisher, 'sub'); + +my $result = + $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl"); + +is($result, qq(5), "check initial copy was done"); + +# Prepare for the next test +$publisher->safe_psql( + 'postgres', qq[ + ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s'; +]); +$publisher->reload; + +$logstart = -s $publisher->logfile; + +# Stop subscriber to make the replication slot on publisher inactive +$subscriber->stop; + +# Wait for the replication slot to become inactive and then invalidated due to +# timeout. +wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart); + +# Testcase end: Invalidate logical subscriber's slot due to inactive timeout +# GUC. +# ============================================================================= + +# ============================================================================= +# Start: Helper functions used for this test file + +sub wait_for_slot_invalidation +{ + my ($node, $slot_name, $offset) = @_; + my $name = $node->name; + + # Wait for the replication slot to become inactive + $node->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND active = 'f'; + ]) + or die + "Timed out while waiting for replication slot to become inactive"; + + # Wait for the replication slot info to be updated + $node->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE inactive_since IS NOT NULL + AND slot_name = '$slot_name' AND active = 'f'; + ]) + or die + "Timed out while waiting for info of replication slot $slot_name to be updated on node $name"; + + check_for_slot_invalidation_in_server_log($node, $slot_name, $offset); + + # Wait for the inactive replication slot to be invalidated. + $node->poll_query_until( + 'postgres', qq[ + SELECT COUNT(slot_name) = 1 FROM pg_replication_slots + WHERE slot_name = '$slot_name' AND + invalidation_reason = 'inactive_timeout'; + ]) + or die + "Timed out while waiting for inactive replication slot $slot_name to be invalidated on node $name"; + + # Check that the invalidated slot cannot be acquired + my ($result, $stdout, $stderr); + + ($result, $stdout, $stderr) = $primary->psql( + 'postgres', qq[ + SELECT pg_replication_slot_advance('$slot_name', '0/1'); + ]); + + ok( $stderr =~ + /can no longer get changes from replication slot "$slot_name"/, + "detected error upon trying to acquire invalidated slot $slot_name on node $name" + ) + or die + "could not detect error upon trying to acquire invalidated slot $slot_name"; +} + +# Check for invalidation of slot in server log. +sub check_for_slot_invalidation_in_server_log +{ + my ($node, $slot_name, $offset) = @_; + my $invalidated = 0; + + for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++) + { + $node->safe_psql('postgres', "CHECKPOINT"); + if ($node->log_contains( + "invalidating obsolete replication slot \"$slot_name\"", + $offset)) + { + $invalidated = 1; + last; + } + usleep(100_000); + } + ok($invalidated, + "check that slot $slot_name invalidation has been logged"); +} + +# ============================================================================= +# End: Helper functions used for this test file + +done_testing(); -- 2.34.1 ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-03 08:17 Bertrand Drouvot <[email protected]> parent: Bharath Rupireddy <[email protected]> 1 sibling, 0 replies; 46+ messages in thread From: Bertrand Drouvot @ 2024-04-03 08:17 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> Hi, On Wed, Apr 03, 2024 at 11:17:41AM +0530, Bharath Rupireddy wrote: > On Wed, Apr 3, 2024 at 8:38 AM shveta malik <[email protected]> wrote: > > > > > > Or a simple solution is that the slotsync worker updates > > > > inactive_since as it does for non-synced slots, and disables > > > > timeout-based slot invalidation for synced slots. > > > > I like this idea better, it takes care of such a case too when the > > user is relying on sync-function rather than worker and does not want > > to get the slots invalidated in between 2 sync function calls. > > Please find the attached v31 patches implementing the above idea: Thanks! Some comments regarding v31-0002: === testing the behavior T1 === > - synced slots don't get invalidated due to inactive timeout because > such slots not considered active at all as they don't perform logical > decoding (of course, they will perform in fast_forward mode to fix the > other data loss issue, but they don't generate changes for them to be > called as *active* slots) It behaves as described. OTOH non synced logical slots on the standby and physical slots on the standby are invalidated which is what is expected. T2 === In case the slot is invalidated on the primary, primary: postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1'; slot_name | inactive_since | invalidation_reason -----------+-------------------------------+--------------------- s1 | 2024-04-03 06:56:28.075637+00 | inactive_timeout then on the standby we get: standby: postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1'; slot_name | inactive_since | invalidation_reason -----------+------------------------------+--------------------- s1 | 2024-04-03 07:06:43.37486+00 | inactive_timeout shouldn't the slot be dropped/recreated instead of updating inactive_since? === code CR1 === + Invalidates replication slots that are inactive for longer the + specified amount of time s/for longer the/for longer that/? CR2 === + <literal>true</literal>) as such synced slots don't actually perform + logical decoding. We're switching in fast forward logical due to [1], so I'm not sure that's 100% accurate here. I'm not sure we need to specify a reason. CR3 === + errdetail("This slot has been invalidated because it was inactive for more than the time specified by replication_slot_inactive_timeout parameter."))); I think we can remove "parameter" (see for example the error message in validate_remote_info()) and reduce it a bit, something like? "This slot has been invalidated because it was inactive for more than replication_slot_inactive_timeout"? CR4 === + appendStringInfoString(&err_detail, _("The slot has been inactive for more than the time specified by replication_slot_inactive_timeout parameter.")); Same. CR5 === + /* + * This function isn't expected to be called for inactive timeout based + * invalidation. A separate function InvalidateInactiveReplicationSlot is + * to be used for that. Do you think it's worth to explain why? CR6 === + if (replication_slot_inactive_timeout == 0) + return false; + else if (slot->inactive_since > 0) "else" is not needed here. CR7 === + SpinLockAcquire(&slot->mutex); + + /* + * Check if the slot needs to be invalidated due to + * replication_slot_inactive_timeout GUC. We do this with the spinlock + * held to avoid race conditions -- for example the inactive_since + * could change, or the slot could be dropped. + */ + now = GetCurrentTimestamp(); We should not call GetCurrentTimestamp() while holding a spinlock. CR8 === +# Testcase start: Invalidate streaming standby's slot as well as logical +# failover slot on primary due to inactive timeout GUC. Also, check the logical s/inactive timeout GUC/replication_slot_inactive_timeout/? CR9 === +# Start: Helper functions used for this test file +# End: Helper functions used for this test file I think that's the first TAP test with this comment. Not saying we should not but why did you feel the need to add those? [1]: https://www.postgresql.org/message-id/[email protected].... Regards, -- Bertrand Drouvot PostgreSQL Contributors Team RDS Open Source Databases Amazon Web Services: https://aws.amazon.com ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-03 09:27 shveta malik <[email protected]> parent: Bharath Rupireddy <[email protected]> 1 sibling, 1 reply; 46+ messages in thread From: shveta malik @ 2024-04-03 09:27 UTC (permalink / raw) To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]> On Wed, Apr 3, 2024 at 11:17 AM Bharath Rupireddy <[email protected]> wrote: > > On Wed, Apr 3, 2024 at 8:38 AM shveta malik <[email protected]> wrote: > > > > > > Or a simple solution is that the slotsync worker updates > > > > inactive_since as it does for non-synced slots, and disables > > > > timeout-based slot invalidation for synced slots. > > > > I like this idea better, it takes care of such a case too when the > > user is relying on sync-function rather than worker and does not want > > to get the slots invalidated in between 2 sync function calls. > > Please find the attached v31 patches implementing the above idea: > Thanks for the patches, please find few comments: v31-001: 1) system-views.sgml: value will get updated after every synchronization from the corresponding remote slot on the primary. --This is confusing. It will be good to rephrase it. 2) update_synced_slots_inactive_since() --May be, we should mention in the header that this function is called only during promotion. 3) 040_standby_failover_slots_sync.pl: We capture inactive_since_on_primary when we do this for the first time at #175 ALTER SUBSCRIPTION regress_mysub1 DISABLE" But we again recreate the sub and disable it at line #280. Do you think we shall get inactive_since_on_primary again here, to be compared with inactive_since_on_new_primary later? v31-002: (I had reviewed v29-002 but missed to post comments, I think these are still applicable) 1) I think replication_slot_inactivity_timeout was recommended here (instead of replication_slot_inactive_timeout, so please give it a thought): https://www.postgresql.org/message-id/202403260739.udlp7lxixktx%40alvherre.pgsql 2) Commit msg: a) "It is often easy for developers to set a timeout of say 1 or 2 or 3 days at slot level, after which the inactive slots get dropped." Shall we say invalidated rather than dropped? b) "To achieve the above, postgres introduces a GUC allowing users set inactive timeout and then a slot stays inactive for this much amount of time it invalidates the slot." Broken sentence. <have not reviewed 002 patch in detail yet> thanks Shveta ^ permalink raw reply [nested|flat] 46+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation @ 2024-04-03 10:49 Amit Kapila <[email protected]> parent: shveta malik <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Amit Kapila @ 2024-04-03 10:49 UTC (permalink / raw) To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]> On Wed, Apr 3, 2024 at 2:58 PM shveta malik <[email protected]> wrote: > > On Wed, Apr 3, 2024 at 11:17 AM Bharath Rupireddy > <[email protected]> wrote: > > > > On Wed, Apr 3, 2024 at 8:38 AM shveta malik <[email protected]> wrote: > > > > > > > > Or a simple solution is that the slotsync worker updates > > > > > inactive_since as it does for non-synced slots, and disables > > > > > timeout-based slot invalidation for synced slots. > > > > > > I like this idea better, it takes care of such a case too when the > > > user is relying on sync-function rather than worker and does not want > > > to get the slots invalidated in between 2 sync function calls. > > > > Please find the attached v31 patches implementing the above idea: > > > > Thanks for the patches, please find few comments: > > v31-001: > > 1) > system-views.sgml: > value will get updated after every synchronization from the > corresponding remote slot on the primary. > > --This is confusing. It will be good to rephrase it. > > 2) > update_synced_slots_inactive_since() > > --May be, we should mention in the header that this function is called > only during promotion. > > 3) 040_standby_failover_slots_sync.pl: > We capture inactive_since_on_primary when we do this for the first time at #175 > ALTER SUBSCRIPTION regress_mysub1 DISABLE" > > But we again recreate the sub and disable it at line #280. > Do you think we shall get inactive_since_on_primary again here, to be > compared with inactive_since_on_new_primary later? > I think so. Few additional comments on tests: 1. +is( $standby1->safe_psql( + 'postgres', + "SELECT '$inactive_since_on_primary'::timestamptz < '$inactive_since_on_standby'::timestamptz AND + '$inactive_since_on_standby'::timestamptz < '$slot_sync_time'::timestamptz;" Shall we do <= check as we are doing in the main function get_slot_inactive_since_value as the time duration is less so it can be the same as well? Similarly, please check other tests. 2. +=item $node->get_slot_inactive_since_value(self, slot_name, reference_time) + +Get inactive_since column value for a given replication slot validating it +against optional reference time. + +=cut + +sub get_slot_inactive_since_value I see that all callers validate against reference time. It is better to name it validate_slot_inactive_since rather than using get_* as the main purpose is to validate the passed value. -- With Regards, Amit Kapila. ^ permalink raw reply [nested|flat] 46+ messages in thread
* [PATCH v47 1/9] Make index_concurrently_create_copy more general @ 2026-03-24 18:02 Álvaro Herrera <[email protected]> 0 siblings, 0 replies; 46+ messages in thread From: Álvaro Herrera @ 2026-03-24 18:02 UTC (permalink / raw) Add a 'boolean concurrent' option, and make it work for both cases. Also rename it to index_create_copy. This allows it to be reused for other purposes -- specifically, for REPACK CONCURRENTLY. With the CONCURRENTLY option, REPACK cannot simply swap the heap file and rebuild its indexes. Instead, it needs to build a separate set of indexes (including system catalog entries) *before* the actual swap, to reduce the time AccessExclusiveLock needs to be held for. This approach is different from what CREATE INDEX CONCURRENTLY does. Per a suggestion from Mihail Nikalayeu. Author: Antonin Houska <[email protected]> Discussion: https://postgr.es/m/41104.1754922120@localhost --- src/backend/catalog/index.c | 39 +++++++++++++++++++------------- src/backend/commands/indexcmds.c | 15 +++++++----- src/backend/nodes/makefuncs.c | 9 ++++---- src/include/catalog/index.h | 7 +++--- src/include/nodes/makefuncs.h | 4 +++- 5 files changed, 43 insertions(+), 31 deletions(-) diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 1ccfa687f05..b86ad73c626 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -1289,17 +1289,17 @@ index_create(Relation heapRelation, } /* - * index_concurrently_create_copy + * index_create_copy * - * Create concurrently an index based on the definition of the one provided by - * caller. The index is inserted into catalogs and needs to be built later - * on. This is called during concurrent reindex processing. + * Create an index based on the definition of the one provided by caller. The + * index is inserted into catalogs. If 'concurrently' is TRUE, it needs to be + * built later on; otherwise it's built immediately. * * "tablespaceOid" is the tablespace to use for this index. */ Oid -index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, - Oid tablespaceOid, const char *newName) +index_create_copy(Relation heapRelation, bool concurrently, + Oid oldIndexId, Oid tablespaceOid, const char *newName) { Relation indexRelation; IndexInfo *oldInfo, @@ -1318,6 +1318,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, List *indexColNames = NIL; List *indexExprs = NIL; List *indexPreds = NIL; + int flags = 0; indexRelation = index_open(oldIndexId, RowExclusiveLock); @@ -1328,7 +1329,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, * Concurrent build of an index with exclusion constraints is not * supported. */ - if (oldInfo->ii_ExclusionOps != NULL) + if (oldInfo->ii_ExclusionOps != NULL && concurrently) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("concurrent index creation for exclusion constraints is not supported"))); @@ -1384,9 +1385,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, } /* - * Build the index information for the new index. Note that rebuild of - * indexes with exclusion constraints is not supported, hence there is no - * need to fill all the ii_Exclusion* fields. + * Build the index information for the new index. */ newInfo = makeIndexInfo(oldInfo->ii_NumIndexAttrs, oldInfo->ii_NumIndexKeyAttrs, @@ -1395,10 +1394,13 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indexPreds, oldInfo->ii_Unique, oldInfo->ii_NullsNotDistinct, - false, /* not ready for inserts */ - true, + !concurrently, /* isready */ + concurrently, /* concurrent */ indexRelation->rd_indam->amsummarizing, - oldInfo->ii_WithoutOverlaps); + oldInfo->ii_WithoutOverlaps, + oldInfo->ii_ExclusionOps, + oldInfo->ii_ExclusionProcs, + oldInfo->ii_ExclusionStrats); /* * Extract the list of column names and the column numbers for the new @@ -1436,6 +1438,9 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, stattargets[i].isnull = isnull; } + if (concurrently) + flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; + /* * Now create the new index. * @@ -1459,7 +1464,7 @@ index_concurrently_create_copy(Relation heapRelation, Oid oldIndexId, indcoloptions->values, stattargets, reloptionsDatum, - INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT, + flags, 0, true, /* allow table to be a system catalog? */ false, /* is_internal? */ @@ -2453,7 +2458,8 @@ BuildIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) @@ -2513,7 +2519,8 @@ BuildDummyIndexInfo(Relation index) indexStruct->indisready, false, index->rd_indam->amsummarizing, - indexStruct->indisexclusion && indexStruct->indisunique); + indexStruct->indisexclusion && indexStruct->indisunique, + NULL, NULL, NULL); /* fill in attribute numbers */ for (i = 0; i < numAtts; i++) diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 373e8234794..4a2d21915b1 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -244,7 +244,8 @@ CheckIndexCompatible(Oid oldId, */ indexInfo = makeIndexInfo(numberOfAttributes, numberOfAttributes, accessMethodId, NIL, NIL, false, false, - false, false, amsummarizing, isWithoutOverlaps); + false, false, amsummarizing, isWithoutOverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); opclassIds = palloc_array(Oid, numberOfAttributes); @@ -931,7 +932,8 @@ DefineIndex(ParseState *pstate, !concurrent, concurrent, amissummarizing, - stmt->iswithoutoverlaps); + stmt->iswithoutoverlaps, + NULL, NULL, NULL); typeIds = palloc_array(Oid, numberOfAttributes); collationIds = palloc_array(Oid, numberOfAttributes); @@ -3989,10 +3991,11 @@ ReindexRelationConcurrently(const ReindexStmt *stmt, Oid relationOid, const Rein tablespaceid = indexRel->rd_rel->reltablespace; /* Create new index definition based on given index */ - newIndexId = index_concurrently_create_copy(heapRel, - idx->indexId, - tablespaceid, - concurrentName); + newIndexId = index_create_copy(heapRel, + true, + idx->indexId, + tablespaceid, + concurrentName); /* * Now open the relation of the new index, a session-level lock is diff --git a/src/backend/nodes/makefuncs.c b/src/backend/nodes/makefuncs.c index 3cd35c5c457..8d23aa917e5 100644 --- a/src/backend/nodes/makefuncs.c +++ b/src/backend/nodes/makefuncs.c @@ -834,7 +834,8 @@ IndexInfo * makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, bool summarizing, - bool withoutoverlaps) + bool withoutoverlaps, Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats) { IndexInfo *n = makeNode(IndexInfo); @@ -863,9 +864,9 @@ makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, n->ii_PredicateState = NULL; /* exclusion constraints */ - n->ii_ExclusionOps = NULL; - n->ii_ExclusionProcs = NULL; - n->ii_ExclusionStrats = NULL; + n->ii_ExclusionOps = exclusion_ops; + n->ii_ExclusionProcs = exclusion_procs; + n->ii_ExclusionStrats = exclusion_strats; /* speculative inserts */ n->ii_UniqueOps = NULL; diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index a38e95bc0eb..ed9e4c37d27 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -101,10 +101,9 @@ extern Oid index_create(Relation heapRelation, #define INDEX_CONSTR_CREATE_REMOVE_OLD_DEPS (1 << 4) #define INDEX_CONSTR_CREATE_WITHOUT_OVERLAPS (1 << 5) -extern Oid index_concurrently_create_copy(Relation heapRelation, - Oid oldIndexId, - Oid tablespaceOid, - const char *newName); +extern Oid index_create_copy(Relation heapRelation, bool concurrently, + Oid oldIndexId, Oid tablespaceOid, + const char *newName); extern void index_concurrently_build(Oid heapRelationId, Oid indexRelationId); diff --git a/src/include/nodes/makefuncs.h b/src/include/nodes/makefuncs.h index bf54d39feb0..40ec249a7a1 100644 --- a/src/include/nodes/makefuncs.h +++ b/src/include/nodes/makefuncs.h @@ -99,7 +99,9 @@ extern IndexInfo *makeIndexInfo(int numattrs, int numkeyattrs, Oid amoid, List *expressions, List *predicates, bool unique, bool nulls_not_distinct, bool isready, bool concurrent, - bool summarizing, bool withoutoverlaps); + bool summarizing, bool withoutoverlaps, + Oid *exclusion_ops, Oid *exclusion_procs, + uint16 *exclusion_strats); extern Node *makeStringConst(char *str, int location); extern DefElem *makeDefElem(char *name, Node *arg, int location); -- 2.47.3 --2fxmamo6mu2qbxgv Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v47-0002-give-options-bitmask-to-table_delete-table_updat.patch" ^ permalink raw reply [nested|flat] 46+ messages in thread
end of thread, other threads:[~2026-03-24 18:02 UTC | newest] Thread overview: 46+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2024-03-26 08:07 Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]> 2024-03-26 08:24 ` Bertrand Drouvot <[email protected]> 2024-03-26 09:47 ` shveta malik <[email protected]> 2024-03-26 10:20 ` Bertrand Drouvot <[email protected]> 2024-03-26 10:47 ` shveta malik <[email protected]> 2024-03-26 11:05 ` Bharath Rupireddy <[email protected]> 2024-03-26 11:19 ` shveta malik <[email protected]> 2024-03-26 12:31 ` Bertrand Drouvot <[email protected]> 2024-03-26 16:29 ` Bharath Rupireddy <[email protected]> 2024-03-26 17:52 ` Bertrand Drouvot <[email protected]> 2024-03-27 04:38 ` Bharath Rupireddy <[email protected]> 2024-03-27 04:52 ` Amit Kapila <[email protected]> 2024-03-27 04:54 ` shveta malik <[email protected]> 2024-03-27 04:56 ` Bharath Rupireddy <[email protected]> 2024-03-27 05:35 ` Bharath Rupireddy <[email protected]> 2024-03-27 06:09 ` shveta malik <[email protected]> 2024-03-27 09:25 ` Bharath Rupireddy <[email protected]> 2024-03-27 10:11 ` Bertrand Drouvot <[email protected]> 2024-03-27 10:13 ` shveta malik <[email protected]> 2024-03-27 12:25 ` Bharath Rupireddy <[email protected]> 2024-03-27 13:24 ` Bertrand Drouvot <[email protected]> 2024-03-27 15:30 ` Bharath Rupireddy <[email protected]> 2024-03-27 16:12 ` Bertrand Drouvot <[email protected]> 2024-03-28 03:46 ` shveta malik <[email protected]> 2024-03-28 09:43 ` Bertrand Drouvot <[email protected]> 2024-03-31 04:55 ` Bharath Rupireddy <[email protected]> 2024-04-01 09:59 ` Bertrand Drouvot <[email protected]> 2024-03-29 04:09 ` Amit Kapila <[email protected]> 2024-03-29 06:19 ` Bertrand Drouvot <[email protected]> 2024-03-29 09:33 ` Amit Kapila <[email protected]> 2024-04-01 03:17 ` Bharath Rupireddy <[email protected]> 2024-04-01 07:18 ` Bertrand Drouvot <[email protected]> 2024-04-02 03:07 ` Masahiko Sawada <[email protected]> 2024-04-02 06:28 ` Bertrand Drouvot <[email protected]> 2024-04-02 07:11 ` Bharath Rupireddy <[email protected]> 2024-04-02 08:33 ` Bertrand Drouvot <[email protected]> 2024-04-03 03:08 ` shveta malik <[email protected]> 2024-04-03 05:47 ` Bharath Rupireddy <[email protected]> 2024-04-03 08:17 ` Bertrand Drouvot <[email protected]> 2024-04-03 09:27 ` shveta malik <[email protected]> 2024-04-03 10:49 ` Amit Kapila <[email protected]> 2024-03-27 05:48 ` Bertrand Drouvot <[email protected]> 2024-03-27 03:31 ` shveta malik <[email protected]> 2024-03-26 12:35 ` Bertrand Drouvot <[email protected]> 2024-03-27 05:25 ` shveta malik <[email protected]> 2026-03-24 18:02 [PATCH v47 1/9] Make index_concurrently_create_copy more general Álvaro Herrera <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox