public inbox for [email protected]
help / color / mirror / Atom feedRe: Introduce XID age and inactive timeout based replication slot invalidation
98+ messages / 12 participants
[nested] [flat]
* Re: Introduce XID age and inactive timeout based replication slot invalidation
@ 2024-04-03 14:58 Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-04 04:11 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
2024-04-04 05:18 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
0 siblings, 3 replies; 98+ messages in thread
From: Bharath Rupireddy @ 2024-04-03 14:58 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Apr 3, 2024 at 6:46 PM Bertrand Drouvot
<[email protected]> wrote:
>
> Just one comment on v32-0001:
>
> +# Synced slot on the standby must get its own inactive_since.
> +is( $standby1->safe_psql(
> + 'postgres',
> + "SELECT '$inactive_since_on_primary'::timestamptz <= '$inactive_since_on_standby'::timestamptz AND
> + '$inactive_since_on_standby'::timestamptz <= '$slot_sync_time'::timestamptz;"
> + ),
> + "t",
> + 'synchronized slot has got its own inactive_since');
> +
>
> By using <= we are not testing that it must get its own inactive_since (as we
> allow them to be equal in the test). I think we should just add some usleep()
> where appropriate and deny equality during the tests on inactive_since.
Thanks. It looks like we can ignore the equality in all of the
inactive_since comparisons. IIUC, all the TAP tests do run with
primary and standbys on the single BF animals. And, it looks like
assigning the inactive_since timestamps to perl variables is giving
the microseconds precision level
(./tmp_check/log/regress_log_040_standby_failover_slots_sync:inactive_since
2024-04-03 14:30:09.691648+00). FWIW, we already have some TAP and SQL
tests relying on stats_reset timestamps without equality. So, I've
left the equality for the inactive_since tests.
> Except for the above, v32-0001 LGTM.
Thanks. Please see the attached v33-0001 patch after removing equality
on inactive_since TAP tests.
On Wed, Apr 3, 2024 at 1:47 PM Bertrand Drouvot
<[email protected]> wrote:
>
> Some comments regarding v31-0002:
>
> === testing the behavior
>
> T1 ===
>
> > - synced slots don't get invalidated due to inactive timeout because
> > such slots not considered active at all as they don't perform logical
> > decoding (of course, they will perform in fast_forward mode to fix the
> > other data loss issue, but they don't generate changes for them to be
> > called as *active* slots)
>
> It behaves as described. OTOH non synced logical slots on the standby and
> physical slots on the standby are invalidated which is what is expected.
Right.
> T2 ===
>
> In case the slot is invalidated on the primary,
>
> primary:
>
> postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1';
> slot_name | inactive_since | invalidation_reason
> -----------+-------------------------------+---------------------
> s1 | 2024-04-03 06:56:28.075637+00 | inactive_timeout
>
> then on the standby we get:
>
> standby:
>
> postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1';
> slot_name | inactive_since | invalidation_reason
> -----------+------------------------------+---------------------
> s1 | 2024-04-03 07:06:43.37486+00 | inactive_timeout
>
> shouldn't the slot be dropped/recreated instead of updating inactive_since?
The sync slots that are invalidated on the primary aren't dropped and
recreated on the standby. There's no point in doing so because
invalidated slots on the primary can't be made useful. However, I
found that the synced slot is acquired and released unnecessarily
after the invalidation_reason is synced from the primary. I added a
skip check in synchronize_one_slot to skip acquiring and releasing the
slot if it's locally found inactive. With this, inactive_since won't
get updated for invalidated sync slots on the standby as we don't
acquire and release the slot.
> === code
>
> CR1 ===
>
> + Invalidates replication slots that are inactive for longer the
> + specified amount of time
>
> s/for longer the/for longer that/?
Fixed.
> CR2 ===
>
> + <literal>true</literal>) as such synced slots don't actually perform
> + logical decoding.
>
> We're switching in fast forward logical due to [1], so I'm not sure that's 100%
> accurate here. I'm not sure we need to specify a reason.
Fixed.
> CR3 ===
>
> + errdetail("This slot has been invalidated because it was inactive for more than the time specified by replication_slot_inactive_timeout parameter.")));
>
> I think we can remove "parameter" (see for example the error message in
> validate_remote_info()) and reduce it a bit, something like?
>
> "This slot has been invalidated because it was inactive for more than replication_slot_inactive_timeout"?
Done.
> CR4 ===
>
> + appendStringInfoString(&err_detail, _("The slot has been inactive for more than the time specified by replication_slot_inactive_timeout parameter."));
>
> Same.
Done. Changed it to "The slot has been inactive for more than
replication_slot_inactive_timeout."
> CR5 ===
>
> + /*
> + * This function isn't expected to be called for inactive timeout based
> + * invalidation. A separate function InvalidateInactiveReplicationSlot is
> + * to be used for that.
>
> Do you think it's worth to explain why?
Hm, I just wanted to point out the actual function here. I modified it
to something like the following, if others feel we don't need that, I
can remove it.
/*
* Use InvalidateInactiveReplicationSlot for inactive timeout based
* invalidation.
*/
> CR6 ===
>
> + if (replication_slot_inactive_timeout == 0)
> + return false;
> + else if (slot->inactive_since > 0)
>
> "else" is not needed here.
Nothing wrong there, but removed.
> CR7 ===
>
> + SpinLockAcquire(&slot->mutex);
> +
> + /*
> + * Check if the slot needs to be invalidated due to
> + * replication_slot_inactive_timeout GUC. We do this with the spinlock
> + * held to avoid race conditions -- for example the inactive_since
> + * could change, or the slot could be dropped.
> + */
> + now = GetCurrentTimestamp();
>
> We should not call GetCurrentTimestamp() while holding a spinlock.
I was thinking why to add up the wait time to acquire
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);. Now that I
moved it up before the spinlock but after the LWLockAcquire.
> CR8 ===
>
> +# Testcase start: Invalidate streaming standby's slot as well as logical
> +# failover slot on primary due to inactive timeout GUC. Also, check the logical
>
> s/inactive timeout GUC/replication_slot_inactive_timeout/?
Done.
> CR9 ===
>
> +# Start: Helper functions used for this test file
> +# End: Helper functions used for this test file
>
> I think that's the first TAP test with this comment. Not saying we should not but
> why did you feel the need to add those?
Hm. Removed.
> [1]: https://www.postgresql.org/message-id/[email protected]....
On Wed, Apr 3, 2024 at 2:58 PM shveta malik <[email protected]> wrote:
>
> v31-002:
> (I had reviewed v29-002 but missed to post comments, I think these
> are still applicable)
>
> 1) I think replication_slot_inactivity_timeout was recommended here
> (instead of replication_slot_inactive_timeout, so please give it a
> thought):
> https://www.postgresql.org/message-id/202403260739.udlp7lxixktx%40alvherre.pgsql
Yeah. It's synonymous with inactive_since. If others have an opinion
to have replication_slot_inactivity_timeout, I'm fine with it.
> 2) Commit msg:
> a)
> "It is often easy for developers to set a timeout of say 1
> or 2 or 3 days at slot level, after which the inactive slots get
> dropped."
>
> Shall we say invalidated rather than dropped?
Right. Done that.
> b)
> "To achieve the above, postgres introduces a GUC allowing users
> set inactive timeout and then a slot stays inactive for this much
> amount of time it invalidates the slot."
>
> Broken sentence.
Reworded it a bit.
Please find the attached v33 patches.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/x-patch] v33-0001-Allow-synced-slots-to-have-their-own-inactive_si.patch (15.1K, ../../CALj2ACUBiRLHF7VYD-0nYVHSvVBSCQ_5OMbP4tE8Ys_rpvQ--A@mail.gmail.com/2-v33-0001-Allow-synced-slots-to-have-their-own-inactive_si.patch)
download | inline diff:
From 03a9e02df871dc86846b79a62a2aaf00e7152f14 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Wed, 3 Apr 2024 14:38:38 +0000
Subject: [PATCH v33 1/2] Allow synced slots to have their own inactive_since.
The slot's inactive_since isn't currently maintained for
synced slots on the standby. The commit a11f330b55 prevents
updating inactive_since with RecoveryInProgress() check in
RestoreSlotFromDisk(). But, the issue is that
RecoveryInProgress() always returns true in
RestoreSlotFromDisk() as 'xlogctl->SharedRecoveryState' is
always 'RECOVERY_STATE_CRASH' at that time. Because of this,
inactive_since is always NULL on a promoted standby for all
synced slots even after server restart.
Above issue led us to a question as to why we can't just let
standby maintain its own inactive_since for synced slots. This is
consistent with any regular slots and also indicates the last
synchronization time of the slot. This approach simplifies things
when compared to just copying inactive_since received from the
remote slot on the primary (for instance, there can exists clock
drift between primary and standby so just copying inactive_since
from the primary slot to the standby sync slot may not represent
the correct value).
This commit does two things:
1) Maintains inactive_since for sync slots whenever the slot is
released just like any other regular slot.
2) Ensures the value is set to current timestamp during the
shutdown of slot sync machinery to help correctly interpret the
time if the standby gets promoted without a restart.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACWLctoiH-pSjWnEpR54q4DED6rw_BRJm5pCx86_Y01MoQ%40mail.gmail.com
---
doc/src/sgml/system-views.sgml | 7 +++
src/backend/replication/logical/slotsync.c | 51 +++++++++++++++
src/backend/replication/slot.c | 22 +++----
src/test/perl/PostgreSQL/Test/Cluster.pm | 31 ++++++++++
src/test/recovery/t/019_replslot_limit.pl | 26 +-------
.../t/040_standby_failover_slots_sync.pl | 62 +++++++++++++++++++
6 files changed, 160 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 3c8dca8ca3..7ed617170f 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2530,6 +2530,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<para>
The time since the slot has become inactive.
<literal>NULL</literal> if the slot is currently being used.
+ Note that for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>), the
+ <structfield>inactive_since</structfield> indicates the last
+ synchronization (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>)
+ time.
</para></entry>
</row>
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 9ac847b780..755bf40a9a 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -150,6 +150,7 @@ typedef struct RemoteSlot
} RemoteSlot;
static void slotsync_failure_callback(int code, Datum arg);
+static void update_synced_slots_inactive_since(void);
/*
* If necessary, update the local synced slot's metadata based on the data
@@ -584,6 +585,11 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* overwriting 'invalidated' flag to remote_slot's value. See
* InvalidatePossiblyObsoleteSlot() where it invalidates slot directly
* if the slot is not acquired by other processes.
+ *
+ * XXX: If it ever turns out that slot acquire/release is costly for
+ * cases when none of the slot property is changed then we can do a
+ * pre-check to ensure that at least one of the slot property is
+ * changed before acquiring the slot.
*/
ReplicationSlotAcquire(remote_slot->name, true);
@@ -1355,6 +1361,48 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
Assert(false);
}
+/*
+ * Update the inactive_since property for synced slots.
+ *
+ * Note that this function is currently called when we shutdown the slot sync
+ * machinery. This helps correctly interpret the inactive_since if the standby
+ * gets promoted without a restart.
+ */
+static void
+update_synced_slots_inactive_since(void)
+{
+ TimestampTz now = 0;
+
+ /* The slot sync worker mustn't be running by now */
+ Assert(SlotSyncCtx->pid == InvalidPid);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ /* Check if it is a synchronized slot */
+ if (s->in_use && s->data.synced)
+ {
+ Assert(SlotIsLogical(s));
+
+ /*
+ * We get the current time beforehand and only once to avoid
+ * system calls while holding the spinlock.
+ */
+ if (now == 0)
+ now = GetCurrentTimestamp();
+
+ SpinLockAcquire(&s->mutex);
+ s->inactive_since = now;
+ SpinLockRelease(&s->mutex);
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+}
+
/*
* Shut down the slot sync worker.
*/
@@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
if (SlotSyncCtx->pid == InvalidPid)
{
SpinLockRelease(&SlotSyncCtx->mutex);
+ update_synced_slots_inactive_since();
return;
}
SpinLockRelease(&SlotSyncCtx->mutex);
@@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
}
SpinLockRelease(&SlotSyncCtx->mutex);
+
+ update_synced_slots_inactive_since();
}
/*
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index d778c0b921..3bddaae022 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -690,13 +690,10 @@ ReplicationSlotRelease(void)
}
/*
- * Set the last inactive time after marking the slot inactive. We don't
- * set it for the slots currently being synced from the primary to the
- * standby because such slots are typically inactive as decoding is not
- * allowed on those.
+ * Set the time since the slot has become inactive. We get the current
+ * time beforehand to avoid system call while holding the spinlock.
*/
- if (!(RecoveryInProgress() && slot->data.synced))
- now = GetCurrentTimestamp();
+ now = GetCurrentTimestamp();
if (slot->data.persistency == RS_PERSISTENT)
{
@@ -2369,16 +2366,11 @@ RestoreSlotFromDisk(const char *name)
slot->active_pid = 0;
/*
- * We set the last inactive time after loading the slot from the disk
- * into memory. Whoever acquires the slot i.e. makes the slot active
- * will reset it. We don't set it for the slots currently being synced
- * from the primary to the standby because such slots are typically
- * inactive as decoding is not allowed on those.
+ * Set the time since the slot has become inactive after loading the
+ * slot from the disk into memory. Whoever acquires the slot i.e.
+ * makes the slot active will reset it.
*/
- if (!(RecoveryInProgress() && slot->data.synced))
- slot->inactive_since = GetCurrentTimestamp();
- else
- slot->inactive_since = 0;
+ slot->inactive_since = GetCurrentTimestamp();
restored = true;
break;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index b08296605c..54e1008ae5 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -3276,6 +3276,37 @@ sub create_logical_slot_on_standby
=pod
+=item $node->validate_slot_inactive_since(self, slot_name, reference_time)
+
+Validate inactive_since value of a given replication slot against the reference
+time and return it.
+
+=cut
+
+sub validate_slot_inactive_since
+{
+ my ($self, $slot_name, $reference_time) = @_;
+ my $name = $self->name;
+
+ my $inactive_since = $self->safe_psql('postgres',
+ qq(SELECT inactive_since FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;)
+ );
+
+ # Check that the inactive_since is sane
+ is($self->safe_psql('postgres',
+ qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND
+ '$inactive_since'::timestamptz > '$reference_time'::timestamptz;]
+ ),
+ 't',
+ "last inactive time for slot $slot_name is valid on node $name")
+ or die "could not validate captured inactive_since for slot $slot_name";
+
+ return $inactive_since;
+}
+
+=pod
+
=item $node->advance_wal(num)
Advance WAL of node by given number of segments.
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index 3b9a306a8b..96b60cedbb 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -443,7 +443,7 @@ $primary4->safe_psql(
# Get inactive_since value after the slot's creation. Note that the slot is
# still inactive till it's used by the standby below.
my $inactive_since =
- capture_and_validate_slot_inactive_since($primary4, $sb4_slot, $slot_creation_time);
+ $primary4->validate_slot_inactive_since($sb4_slot, $slot_creation_time);
$standby4->start;
@@ -502,7 +502,7 @@ $publisher4->safe_psql('postgres',
# Get inactive_since value after the slot's creation. Note that the slot is
# still inactive till it's used by the subscriber below.
$inactive_since =
- capture_and_validate_slot_inactive_since($publisher4, $lsub4_slot, $slot_creation_time);
+ $publisher4->validate_slot_inactive_since($lsub4_slot, $slot_creation_time);
$subscriber4->start;
$subscriber4->safe_psql('postgres',
@@ -540,26 +540,4 @@ is( $publisher4->safe_psql(
$publisher4->stop;
$subscriber4->stop;
-# Capture and validate inactive_since of a given slot.
-sub capture_and_validate_slot_inactive_since
-{
- my ($node, $slot_name, $slot_creation_time) = @_;
-
- my $inactive_since = $node->safe_psql('postgres',
- qq(SELECT inactive_since FROM pg_replication_slots
- WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;)
- );
-
- # Check that the captured time is sane
- is( $node->safe_psql(
- 'postgres',
- qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND
- '$inactive_since'::timestamptz >= '$slot_creation_time'::timestamptz;]
- ),
- 't',
- "last inactive time for an active slot $slot_name is sane");
-
- return $inactive_since;
-}
-
done_testing();
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 869e3d2e91..e7d92e3276 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1');
$subscriber1->init;
$subscriber1->start;
+# Capture the time before the logical failover slot is created on the
+# primary. We later call this publisher as primary anyway.
+my $slot_creation_time_on_primary = $publisher->safe_psql(
+ 'postgres', qq[
+ SELECT current_timestamp;
+]);
+
# Create a slot on the publisher with failover disabled
$publisher->safe_psql('postgres',
"SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);"
@@ -174,6 +181,11 @@ $primary->poll_query_until(
"SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
1);
+# Capture the inactive_since of the slot from the primary. Note that the slot
+# is not yet active.
+my $inactive_since_on_primary =
+ $primary->validate_slot_inactive_since('lsub1_slot', $slot_creation_time_on_primary);
+
# Wait for the standby to catch up so that the standby is not lagging behind
# the subscriber.
$primary->wait_for_replay_catchup($standby1);
@@ -181,6 +193,11 @@ $primary->wait_for_replay_catchup($standby1);
# Synchronize the primary server slots to the standby.
$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+my $slot_sync_time = $standby1->safe_psql(
+ 'postgres', qq[
+ SELECT current_timestamp;
+]);
+
# Confirm that the logical failover slots are created on the standby and are
# flagged as 'synced'
is( $standby1->safe_psql(
@@ -190,6 +207,19 @@ is( $standby1->safe_psql(
"t",
'logical slots have synced as true on standby');
+# Capture the inactive_since of the synced slot on the standby
+my $inactive_since_on_standby =
+ $standby1->validate_slot_inactive_since('lsub1_slot', $slot_creation_time_on_primary);
+
+# Synced slot on the standby must get its own inactive_since.
+is( $standby1->safe_psql(
+ 'postgres',
+ "SELECT '$inactive_since_on_primary'::timestamptz < '$inactive_since_on_standby'::timestamptz AND
+ '$inactive_since_on_standby'::timestamptz < '$slot_sync_time'::timestamptz;"
+ ),
+ "t",
+ 'synchronized slot has got its own inactive_since');
+
##################################################
# Test that the synchronized slot will be dropped if the corresponding remote
# slot on the primary server has been dropped.
@@ -237,6 +267,13 @@ is( $standby1->safe_psql(
$standby1->append_conf('postgresql.conf', 'max_slot_wal_keep_size = -1');
$standby1->reload;
+# Capture the time before the logical failover slot is created on the primary.
+# Note that the subscription creates the slot again on the primary.
+$slot_creation_time_on_primary = $publisher->safe_psql(
+ 'postgres', qq[
+ SELECT current_timestamp;
+]);
+
# To ensure that restart_lsn has moved to a recent WAL position, we re-create
# the subscription and the logical slot.
$subscriber1->safe_psql(
@@ -257,6 +294,11 @@ $primary->poll_query_until(
"SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
1);
+# Capture the inactive_since of the slot from the primary. Note that the slot
+# is not yet active but has been dropped and recreated.
+$inactive_since_on_primary =
+ $primary->validate_slot_inactive_since('lsub1_slot', $slot_creation_time_on_primary);
+
# Wait for the standby to catch up so that the standby is not lagging behind
# the subscriber.
$primary->wait_for_replay_catchup($standby1);
@@ -808,8 +850,28 @@ $primary->reload;
$standby1->start;
$primary->wait_for_replay_catchup($standby1);
+# Capture the time before the standby is promoted
+my $promotion_time_on_primary = $standby1->safe_psql(
+ 'postgres', qq[
+ SELECT current_timestamp;
+]);
+
$standby1->promote;
+# Capture the inactive_since of the synced slot after the promotion.
+# Expectation here is that the slot gets its own inactive_since as part of the
+# promotion. We do this check before the slot is enabled on the new primary
+# below, otherwise the slot gets active setting inactive_since to NULL.
+my $inactive_since_on_new_primary =
+ $standby1->validate_slot_inactive_since('lsub1_slot', $promotion_time_on_primary);
+
+is( $standby1->safe_psql(
+ 'postgres',
+ "SELECT '$inactive_since_on_new_primary'::timestamptz > '$inactive_since_on_primary'::timestamptz"
+ ),
+ "t",
+ 'synchronized slot has got its own inactive_since on the new primary after promotion');
+
# Update subscription with the new primary's connection info
my $standby1_conninfo = $standby1->connstr . ' dbname=postgres';
$subscriber1->safe_psql('postgres',
--
2.34.1
[application/x-patch] v33-0002-Add-inactive_timeout-based-replication-slot-inva.patch (32.1K, ../../CALj2ACUBiRLHF7VYD-0nYVHSvVBSCQ_5OMbP4tE8Ys_rpvQ--A@mail.gmail.com/3-v33-0002-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From abe5113e8431e0a691ef61c04288dea6a0f4a2ac Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Wed, 3 Apr 2024 14:39:36 +0000
Subject: [PATCH v33 2/2] Add inactive_timeout based replication slot
invalidation.
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set a timeout of say 1
or 2 or 3 days at slot level, after which the inactive slots get
invalidated.
To achieve the above, postgres introduces a GUC allowing users
set inactive timeout. The replication slots that are inactive
for longer than specified amount of time get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Note that this new invalidation mechanism won't kick-in for the
slots that are currently being synced from the primary to the
standby.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
---
doc/src/sgml/config.sgml | 32 ++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 11 +-
src/backend/replication/slot.c | 208 ++++++++++++-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 8 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/044_invalidate_slots.pl | 274 ++++++++++++++++++
13 files changed, 540 insertions(+), 24 deletions(-)
create mode 100644 src/test/recovery/t/044_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 624518e0b0..626eac7125 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4547,6 +4547,38 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
+ <para>
+ The timeout is measured from the time since the slot has become
+ inactive (known from its
+ <structfield>inactive_since</structfield> value) until it gets
+ used (i.e., its <structfield>active</structfield> is set to true).
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 7ed617170f..063638beda 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2580,6 +2580,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for the duration specified by
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 755bf40a9a..080edc0d74 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -362,7 +362,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -575,6 +575,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
" name slot \"%s\" already exists on the standby",
remote_slot->name));
+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to save on slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
+
/*
* The slot has been synchronized before.
*
@@ -591,7 +598,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot property is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 3bddaae022..e5ee934b24 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -158,6 +160,7 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyInactiveSlot(ReplicationSlot *slot);
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
@@ -535,9 +538,14 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * If check_for_timeout_invalidation is true, the slot is checked for
+ * invalidation based on replication_slot_inactive_timeout GUC, and an error is
+ * raised after making the slot ours.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_timeout_invalidation)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +623,34 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * Check if the given slot can be invalidated based on its inactive
+ * timeout. If yes, persist the invalidated state to disk and then error
+ * out. We do this only after making the slot ours to avoid anyone else
+ * acquiring it while we check for its invalidation.
+ */
+ if (check_for_timeout_invalidation)
+ {
+ /* The slot is ours by now */
+ Assert(s->active_pid == MyProcPid);
+
+ if (InvalidateInactiveReplicationSlot(s, true))
+ {
+ /*
+ * If the slot has been invalidated, recalculate the resource
+ * limits.
+ */
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(MyReplicationSlot->data.name)),
+ errdetail("This slot has been invalidated because it was inactive for more than replication_slot_inactive_timeout.")));
+ }
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -781,7 +817,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -804,7 +840,7 @@ ReplicationSlotAlter(const char *name, bool failover)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -980,6 +1016,20 @@ ReplicationSlotDropPtr(ReplicationSlot *slot)
LWLockRelease(ReplicationSlotAllocationLock);
}
+/*
+ * Helper for ReplicationSlotSave
+ */
+static inline void
+SaveGivenReplicationSlot(ReplicationSlot *slot, int elevel)
+{
+ char path[MAXPGPATH];
+
+ Assert(slot != NULL);
+
+ sprintf(path, "pg_replslot/%s", NameStr(slot->data.name));
+ SaveSlotToPath(slot, path, elevel);
+}
+
/*
* Serialize the currently acquired slot's state from memory to disk, thereby
* guaranteeing the current state will survive a crash.
@@ -987,12 +1037,21 @@ ReplicationSlotDropPtr(ReplicationSlot *slot)
void
ReplicationSlotSave(void)
{
- char path[MAXPGPATH];
+ SaveGivenReplicationSlot(MyReplicationSlot, ERROR);
+}
- Assert(MyReplicationSlot != NULL);
+/*
+ * Helper for ReplicationSlotMarkDirty
+ */
+static inline void
+MarkGivenReplicationSlotDirty(ReplicationSlot *slot)
+{
+ Assert(slot != NULL);
- sprintf(path, "pg_replslot/%s", NameStr(MyReplicationSlot->data.name));
- SaveSlotToPath(MyReplicationSlot, path, ERROR);
+ SpinLockAcquire(&slot->mutex);
+ slot->just_dirtied = true;
+ slot->dirty = true;
+ SpinLockRelease(&slot->mutex);
}
/*
@@ -1005,14 +1064,7 @@ ReplicationSlotSave(void)
void
ReplicationSlotMarkDirty(void)
{
- ReplicationSlot *slot = MyReplicationSlot;
-
- Assert(MyReplicationSlot != NULL);
-
- SpinLockAcquire(&slot->mutex);
- MyReplicationSlot->just_dirtied = true;
- MyReplicationSlot->dirty = true;
- SpinLockRelease(&slot->mutex);
+ MarkGivenReplicationSlotDirty(MyReplicationSlot);
}
/*
@@ -1506,6 +1558,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires wal_level >= logical on the primary server."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ appendStringInfoString(&err_detail, _("The slot has been inactive for more than replication_slot_inactive_timeout."));
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1550,6 +1605,12 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ /*
+ * Use InvalidateInactiveReplicationSlot for inactive timeout based
+ * invalidation.
+ */
+ Assert(cause != RS_INVAL_INACTIVE_TIMEOUT);
+
for (;;)
{
XLogRecPtr restart_lsn;
@@ -1619,6 +1680,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ /* not reachable */
+ Assert(false);
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1772,6 +1837,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout occurs
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1787,6 +1853,12 @@ InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause cause,
Assert(cause != RS_INVAL_WAL_REMOVED || oldestSegno > 0);
Assert(cause != RS_INVAL_NONE);
+ /*
+ * Use InvalidateInactiveReplicationSlot for inactive timeout based
+ * invalidation.
+ */
+ Assert(cause != RS_INVAL_INACTIVE_TIMEOUT);
+
if (max_replication_slots == 0)
return invalidated;
@@ -1823,6 +1895,95 @@ restart:
return invalidated;
}
+/*
+ * Invalidate given slot based on replication_slot_inactive_timeout GUC.
+ *
+ * Returns true if the slot has got invalidated.
+ *
+ * NB - this function also runs as part of checkpoint, so avoid raising errors
+ * if possible.
+ */
+bool
+InvalidateInactiveReplicationSlot(ReplicationSlot *slot, bool persist_state)
+{
+ if (!InvalidatePossiblyInactiveSlot(slot))
+ return false;
+
+ /* Make sure the invalidated state persists across server restart */
+ MarkGivenReplicationSlotDirty(slot);
+
+ if (persist_state)
+ SaveGivenReplicationSlot(slot, ERROR);
+
+ ReportSlotInvalidation(RS_INVAL_INACTIVE_TIMEOUT, false, 0,
+ slot->data.name, InvalidXLogRecPtr,
+ InvalidXLogRecPtr, InvalidTransactionId);
+
+ return true;
+}
+
+/*
+ * Helper for InvalidateInactiveReplicationSlot
+ */
+static bool
+InvalidatePossiblyInactiveSlot(ReplicationSlot *slot)
+{
+ ReplicationSlotInvalidationCause inavidation_cause = RS_INVAL_NONE;
+
+ /*
+ * Note that we don't invalidate slot on the standby that's currently
+ * being synced from the primary, because such slots are typically
+ * considered not active as they don't actually perform logical decoding.
+ */
+ if (RecoveryInProgress() && slot->data.synced)
+ return false;
+
+ if (replication_slot_inactive_timeout == 0)
+ return false;
+
+ if (slot->inactive_since > 0)
+ {
+ TimestampTz now;
+
+ /*
+ * Do not invalidate the slots which are currently being synced from
+ * the primary to the standby.
+ */
+ if (RecoveryInProgress() && slot->data.synced)
+ return false;
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+
+ SpinLockAcquire(&slot->mutex);
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC. We do this with the spinlock
+ * held to avoid race conditions -- for example the inactive_since
+ * could change, or the slot could be dropped.
+ */
+ if (TimestampDifferenceExceeds(slot->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ inavidation_cause = RS_INVAL_INACTIVE_TIMEOUT;
+ slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT;
+ }
+
+ SpinLockRelease(&slot->mutex);
+ LWLockRelease(ReplicationSlotControlLock);
+
+ return (inavidation_cause == RS_INVAL_INACTIVE_TIMEOUT);
+ }
+
+ return false;
+}
+
/*
* Flush all replication slots to disk.
*
@@ -1835,6 +1996,7 @@ void
CheckPointReplicationSlots(bool is_shutdown)
{
int i;
+ bool invalidated = false;
elog(DEBUG1, "performing replication slot checkpoint");
@@ -1858,6 +2020,13 @@ CheckPointReplicationSlots(bool is_shutdown)
/* save the slot to disk, locking is handled in SaveSlotToPath() */
sprintf(path, "pg_replslot/%s", NameStr(s->data.name));
+ /*
+ * Here's an opportunity to invalidate inactive replication slots
+ * based on timeout, so let's do it.
+ */
+ if (InvalidateInactiveReplicationSlot(s, false))
+ invalidated = true;
+
/*
* Slot's data is not flushed each time the confirmed_flush LSN is
* updated as that could lead to frequent writes. However, we decide
@@ -1884,6 +2053,13 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ /* If the slot has been invalidated, recalculate the resource limits */
+ if (invalidated)
+ {
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index dd6c1d5a7e..9ad3e55704 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -539,7 +539,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index bc40c454de..96eeb8b7d2 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index c12784cbec..4149ff1ffe 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2971,6 +2971,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index baecde2841..2e1ad2eaca 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 7b937d1a0c..f0ac324ce9 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *standby_slot_names;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -245,7 +248,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait);
extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, bool failover);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_timeout_invalidation);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(void);
extern void ReplicationSlotSave(void);
@@ -264,6 +268,8 @@ extern bool InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause
XLogSegNo oldestSegno,
Oid dboid,
TransactionId snapshotConflictHorizon);
+extern bool InvalidateInactiveReplicationSlot(ReplicationSlot *slot,
+ bool persist_state);
extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_lock);
extern int ReplicationSlotIndex(ReplicationSlot *slot);
extern bool ReplicationSlotName(int index, Name name);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 712924c2fa..0437ab5c46 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -52,6 +52,7 @@ tests += {
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
't/043_wal_replay_wait.pl',
+ 't/044_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/044_invalidate_slots.pl b/src/test/recovery/t/044_invalidate_slots.pl
new file mode 100644
index 0000000000..8f7967a253
--- /dev/null
+++ b/src/test/recovery/t/044_invalidate_slots.pl
@@ -0,0 +1,274 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to replication_slot_inactive_timeout. Also,
+# check the logical failover slot synced on to the standby doesn't invalidate
+# the slot on its own, but gets the invalidated state from the remote slot on
+# the primary.
+
+# Initialize primary node
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid checkpoint during the test, otherwise, the test can get unpredictable
+$primary->append_conf(
+ 'postgresql.conf', q{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create a standby linking to the primary using the replication slot
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr_1 = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb1_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot');
+]);
+
+$standby1->start;
+
+my $standby1_logstart = -s $standby1->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Synchronize the primary server slots to the standby.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced'.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot has synced as true on standby');
+
+my $logstart = -s $primary->logfile;
+
+# Set timeout so that the next checkpoint will invalidate the inactive
+# replication slot.
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$primary->reload;
+
+# Wait for the logical failover slot to become inactive on the primary. Note
+# that nobody has acquired that slot yet, so due to
+# replication_slot_inactive_timeout setting above it must get invalidated.
+wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart);
+
+# Set timeout on the standby also to check the synced slots don't get
+# invalidated due to timeout on the standby.
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$standby1->reload;
+
+# Now, sync the logical failover slot from the remote slot on the primary.
+# Note that the remote slot has already been invalidated due to inactive
+# timeout. Now, the standby must also see it as invalidated.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby";
+
+# Synced slot mustn't get invalidated on the standby, it must sync invalidation
+# from the primary. So, we must not see the slot's invalidation message in server
+# log.
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned slot has not been invalidated on the standby');
+
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart);
+
+# Testcase end: Invalidate streaming standby's slot as well as logical failover
+# slot on primary due to replication_slot_inactive_timeout. Also, check the
+# logical failover slot synced on to the standby doesn't invalidate the slot on
+# its own, but gets the invalidated state from the remote slot on the primary.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+
+my $publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber node
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$subscriber->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart);
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
+
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $name = $node->name;
+
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for replication slot to become inactive";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of replication slot $slot_name to be updated on node $name";
+
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+
+ # Wait for the inactive replication slot to be invalidated.
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive replication slot $slot_name to be invalidated on node $name";
+
+ # Check that the invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $primary->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot_name', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot_name"/,
+ "detected error upon trying to acquire invalidated slot $slot_name on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot_name";
+}
+
+# Check for invalidation of slot in server log.
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot_name\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot_name invalidation has been logged");
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-03 16:27 ` Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2 siblings, 1 reply; 98+ messages in thread
From: Bertrand Drouvot @ 2024-04-03 16:27 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On Wed, Apr 03, 2024 at 08:28:04PM +0530, Bharath Rupireddy wrote:
> On Wed, Apr 3, 2024 at 6:46 PM Bertrand Drouvot
> <[email protected]> wrote:
> >
> > Just one comment on v32-0001:
> >
> > +# Synced slot on the standby must get its own inactive_since.
> > +is( $standby1->safe_psql(
> > + 'postgres',
> > + "SELECT '$inactive_since_on_primary'::timestamptz <= '$inactive_since_on_standby'::timestamptz AND
> > + '$inactive_since_on_standby'::timestamptz <= '$slot_sync_time'::timestamptz;"
> > + ),
> > + "t",
> > + 'synchronized slot has got its own inactive_since');
> > +
> >
> > By using <= we are not testing that it must get its own inactive_since (as we
> > allow them to be equal in the test). I think we should just add some usleep()
> > where appropriate and deny equality during the tests on inactive_since.
>
> > Except for the above, v32-0001 LGTM.
>
> Thanks. Please see the attached v33-0001 patch after removing equality
> on inactive_since TAP tests.
Thanks! v33-0001 LGTM.
> On Wed, Apr 3, 2024 at 1:47 PM Bertrand Drouvot
> <[email protected]> wrote:
> > Some comments regarding v31-0002:
> >
> > T2 ===
> >
> > In case the slot is invalidated on the primary,
> >
> > primary:
> >
> > postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1';
> > slot_name | inactive_since | invalidation_reason
> > -----------+-------------------------------+---------------------
> > s1 | 2024-04-03 06:56:28.075637+00 | inactive_timeout
> >
> > then on the standby we get:
> >
> > standby:
> >
> > postgres=# select slot_name, inactive_since, invalidation_reason from pg_replication_slots where slot_name = 's1';
> > slot_name | inactive_since | invalidation_reason
> > -----------+------------------------------+---------------------
> > s1 | 2024-04-03 07:06:43.37486+00 | inactive_timeout
> >
> > shouldn't the slot be dropped/recreated instead of updating inactive_since?
>
> The sync slots that are invalidated on the primary aren't dropped and
> recreated on the standby.
Yeah, right (I was confused with synced slots that are invalidated locally).
> However, I
> found that the synced slot is acquired and released unnecessarily
> after the invalidation_reason is synced from the primary. I added a
> skip check in synchronize_one_slot to skip acquiring and releasing the
> slot if it's locally found inactive. With this, inactive_since won't
> get updated for invalidated sync slots on the standby as we don't
> acquire and release the slot.
CR1 ===
Yeah, I can see:
@@ -575,6 +575,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
" name slot \"%s\" already exists on the standby",
remote_slot->name));
+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to save on slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
Thanks to the drop_local_obsolete_slots() call I think we are not missing the case
where the slot has been invalidated on the primary, invalidation reason has been
synced on the standby and later the slot is dropped/ recreated manually on the
primary (then it should be dropped/recreated on the standby too).
Also it seems we are not missing the case where a sync slot is invalidated
locally due to wal removal (it should be dropped/recreated).
>
> > CR5 ===
> >
> > + /*
> > + * This function isn't expected to be called for inactive timeout based
> > + * invalidation. A separate function InvalidateInactiveReplicationSlot is
> > + * to be used for that.
> >
> > Do you think it's worth to explain why?
>
> Hm, I just wanted to point out the actual function here. I modified it
> to something like the following, if others feel we don't need that, I
> can remove it.
Sorry If I was not clear but I meant to say "Do you think it's worth to explain
why we decided to create a dedicated function"? (currently we "just" explain that
we created one).
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
@ 2024-04-05 05:51 ` Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-04-05 05:51 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Apr 3, 2024 at 9:57 PM Bertrand Drouvot
<[email protected]> wrote:
>
> > > shouldn't the slot be dropped/recreated instead of updating inactive_since?
> >
> > The sync slots that are invalidated on the primary aren't dropped and
> > recreated on the standby.
>
> Yeah, right (I was confused with synced slots that are invalidated locally).
>
> > However, I
> > found that the synced slot is acquired and released unnecessarily
> > after the invalidation_reason is synced from the primary. I added a
> > skip check in synchronize_one_slot to skip acquiring and releasing the
> > slot if it's locally found inactive. With this, inactive_since won't
> > get updated for invalidated sync slots on the standby as we don't
> > acquire and release the slot.
>
> CR1 ===
>
> Yeah, I can see:
>
> @@ -575,6 +575,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
> " name slot \"%s\" already exists on the standby",
> remote_slot->name));
>
> + /*
> + * Skip the sync if the local slot is already invalidated. We do this
> + * beforehand to save on slot acquire and release.
> + */
> + if (slot->data.invalidated != RS_INVAL_NONE)
> + return false;
>
> Thanks to the drop_local_obsolete_slots() call I think we are not missing the case
> where the slot has been invalidated on the primary, invalidation reason has been
> synced on the standby and later the slot is dropped/ recreated manually on the
> primary (then it should be dropped/recreated on the standby too).
>
> Also it seems we are not missing the case where a sync slot is invalidated
> locally due to wal removal (it should be dropped/recreated).
Right.
> > > CR5 ===
> > >
> > > + /*
> > > + * This function isn't expected to be called for inactive timeout based
> > > + * invalidation. A separate function InvalidateInactiveReplicationSlot is
> > > + * to be used for that.
> > >
> > > Do you think it's worth to explain why?
> >
> > Hm, I just wanted to point out the actual function here. I modified it
> > to something like the following, if others feel we don't need that, I
> > can remove it.
>
> Sorry If I was not clear but I meant to say "Do you think it's worth to explain
> why we decided to create a dedicated function"? (currently we "just" explain that
> we created one).
We added a new function (InvalidateInactiveReplicationSlot) to
invalidate slot based on inactive timeout because 1) we do the
inactive timeout invalidation at slot level as opposed to
InvalidateObsoleteReplicationSlots which does loop over all the slots,
2)
InvalidatePossiblyObsoleteSlot does release the lock in some cases,
has a lot of unneeded code for inactive timeout invalidation check, 3)
we want some control over saving the slot to disk because we hook the
inactive timeout invalidation into the loop that checkpoints the slot
info to the disk in CheckPointReplicationSlots.
I've added a comment atop InvalidateInactiveReplicationSlot.
Please find the attached v36 patch.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v36-0001-Add-inactive_timeout-based-replication-slot-inva.patch (32.1K, ../../CALj2ACXbUzp-9tixeUK_dws5k+ARJaXxj8cj9W3adA0XNUS4Hg@mail.gmail.com/2-v36-0001-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From 1410136f2d7a5b6cf0b52dba63ded96f177046a7 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Fri, 5 Apr 2024 05:48:14 +0000
Subject: [PATCH v36] Add inactive_timeout based replication slot invalidation.
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set a timeout of say 1
or 2 or 3 days at slot level, after which the inactive slots get
invalidated.
To achieve the above, postgres introduces a GUC allowing users
set inactive timeout. The replication slots that are inactive
for longer than specified amount of time get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Note that this new invalidation mechanism won't kick-in for the
slots that are currently being synced from the primary to the
standby.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
---
doc/src/sgml/config.sgml | 32 ++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 11 +-
src/backend/replication/slot.c | 215 +++++++++++++-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 8 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/044_invalidate_slots.pl | 274 ++++++++++++++++++
13 files changed, 547 insertions(+), 24 deletions(-)
create mode 100644 src/test/recovery/t/044_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 624518e0b0..626eac7125 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4547,6 +4547,38 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
+ <para>
+ The timeout is measured from the time since the slot has become
+ inactive (known from its
+ <structfield>inactive_since</structfield> value) until it gets
+ used (i.e., its <structfield>active</structfield> is set to true).
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 7ed617170f..063638beda 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2580,6 +2580,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for the duration specified by
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index d18e2c7342..e92f559539 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -362,7 +362,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -575,6 +575,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
" name slot \"%s\" already exists on the standby",
remote_slot->name));
+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to avoid slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
+
/*
* The slot has been synchronized before.
*
@@ -591,7 +598,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 3bddaae022..7006c2e2fd 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -158,6 +160,7 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyInactiveSlot(ReplicationSlot *slot);
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
@@ -535,9 +538,14 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * If check_for_timeout_invalidation is true, the slot is checked for
+ * invalidation based on replication_slot_inactive_timeout GUC, and an error is
+ * raised after making the slot ours.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_timeout_invalidation)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +623,34 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * Check if the given slot can be invalidated based on its inactive
+ * timeout. If yes, persist the invalidated state to disk and then error
+ * out. We do this only after making the slot ours to avoid anyone else
+ * acquiring it while we check for its invalidation.
+ */
+ if (check_for_timeout_invalidation)
+ {
+ /* The slot is ours by now */
+ Assert(s->active_pid == MyProcPid);
+
+ if (InvalidateInactiveReplicationSlot(s, true))
+ {
+ /*
+ * If the slot has been invalidated, recalculate the resource
+ * limits.
+ */
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(MyReplicationSlot->data.name)),
+ errdetail("This slot has been invalidated because it was inactive for more than replication_slot_inactive_timeout.")));
+ }
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -781,7 +817,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -804,7 +840,7 @@ ReplicationSlotAlter(const char *name, bool failover)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -980,6 +1016,20 @@ ReplicationSlotDropPtr(ReplicationSlot *slot)
LWLockRelease(ReplicationSlotAllocationLock);
}
+/*
+ * Helper for ReplicationSlotSave
+ */
+static inline void
+SaveGivenReplicationSlot(ReplicationSlot *slot, int elevel)
+{
+ char path[MAXPGPATH];
+
+ Assert(slot != NULL);
+
+ sprintf(path, "pg_replslot/%s", NameStr(slot->data.name));
+ SaveSlotToPath(slot, path, elevel);
+}
+
/*
* Serialize the currently acquired slot's state from memory to disk, thereby
* guaranteeing the current state will survive a crash.
@@ -987,12 +1037,21 @@ ReplicationSlotDropPtr(ReplicationSlot *slot)
void
ReplicationSlotSave(void)
{
- char path[MAXPGPATH];
+ SaveGivenReplicationSlot(MyReplicationSlot, ERROR);
+}
- Assert(MyReplicationSlot != NULL);
+/*
+ * Helper for ReplicationSlotMarkDirty
+ */
+static inline void
+MarkGivenReplicationSlotDirty(ReplicationSlot *slot)
+{
+ Assert(slot != NULL);
- sprintf(path, "pg_replslot/%s", NameStr(MyReplicationSlot->data.name));
- SaveSlotToPath(MyReplicationSlot, path, ERROR);
+ SpinLockAcquire(&slot->mutex);
+ slot->just_dirtied = true;
+ slot->dirty = true;
+ SpinLockRelease(&slot->mutex);
}
/*
@@ -1005,14 +1064,7 @@ ReplicationSlotSave(void)
void
ReplicationSlotMarkDirty(void)
{
- ReplicationSlot *slot = MyReplicationSlot;
-
- Assert(MyReplicationSlot != NULL);
-
- SpinLockAcquire(&slot->mutex);
- MyReplicationSlot->just_dirtied = true;
- MyReplicationSlot->dirty = true;
- SpinLockRelease(&slot->mutex);
+ MarkGivenReplicationSlotDirty(MyReplicationSlot);
}
/*
@@ -1506,6 +1558,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires wal_level >= logical on the primary server."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ appendStringInfoString(&err_detail, _("The slot has been inactive for more than replication_slot_inactive_timeout."));
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1550,6 +1605,12 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ /*
+ * Use InvalidateInactiveReplicationSlot for inactive timeout based
+ * invalidation.
+ */
+ Assert(cause != RS_INVAL_INACTIVE_TIMEOUT);
+
for (;;)
{
XLogRecPtr restart_lsn;
@@ -1619,6 +1680,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ /* not reachable */
+ Assert(false);
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1787,6 +1852,12 @@ InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause cause,
Assert(cause != RS_INVAL_WAL_REMOVED || oldestSegno > 0);
Assert(cause != RS_INVAL_NONE);
+ /*
+ * Use InvalidateInactiveReplicationSlot for inactive timeout based
+ * invalidation.
+ */
+ Assert(cause != RS_INVAL_INACTIVE_TIMEOUT);
+
if (max_replication_slots == 0)
return invalidated;
@@ -1823,6 +1894,103 @@ restart:
return invalidated;
}
+/*
+ * Invalidate given slot based on replication_slot_inactive_timeout GUC.
+ *
+ * Returns true if the slot has got invalidated.
+ *
+ * Whether the given slot needs to be invalidated depends on the cause:
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout occurs
+ *
+ * NB - this function also runs as part of checkpoint, so avoid raising errors
+ * if possible.
+ *
+ * Note that having a new function for RS_INVAL_INACTIVE_TIMEOUT cause instead
+ * of using InvalidateObsoleteReplicationSlots provides flexibility in calling
+ * it at slot-level opportunistically, and choosing to persist the slot info to
+ * disk.
+ */
+bool
+InvalidateInactiveReplicationSlot(ReplicationSlot *slot, bool persist_state)
+{
+ if (!InvalidatePossiblyInactiveSlot(slot))
+ return false;
+
+ /* Make sure the invalidated state persists across server restart */
+ MarkGivenReplicationSlotDirty(slot);
+
+ if (persist_state)
+ SaveGivenReplicationSlot(slot, ERROR);
+
+ ReportSlotInvalidation(RS_INVAL_INACTIVE_TIMEOUT, false, 0,
+ slot->data.name, InvalidXLogRecPtr,
+ InvalidXLogRecPtr, InvalidTransactionId);
+
+ return true;
+}
+
+/*
+ * Helper for InvalidateInactiveReplicationSlot
+ */
+static bool
+InvalidatePossiblyInactiveSlot(ReplicationSlot *slot)
+{
+ ReplicationSlotInvalidationCause inavidation_cause = RS_INVAL_NONE;
+
+ /*
+ * Note that we don't invalidate slot on the standby that's currently
+ * being synced from the primary, because such slots are typically
+ * considered not active as they don't actually perform logical decoding.
+ */
+ if (RecoveryInProgress() && slot->data.synced)
+ return false;
+
+ if (replication_slot_inactive_timeout == 0)
+ return false;
+
+ if (slot->inactive_since > 0)
+ {
+ TimestampTz now;
+
+ /*
+ * Do not invalidate the slots which are currently being synced from
+ * the primary to the standby.
+ */
+ if (RecoveryInProgress() && slot->data.synced)
+ return false;
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+
+ SpinLockAcquire(&slot->mutex);
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC. We do this with the spinlock
+ * held to avoid race conditions -- for example the inactive_since
+ * could change, or the slot could be dropped.
+ */
+ if (TimestampDifferenceExceeds(slot->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ inavidation_cause = RS_INVAL_INACTIVE_TIMEOUT;
+ slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT;
+ }
+
+ SpinLockRelease(&slot->mutex);
+ LWLockRelease(ReplicationSlotControlLock);
+
+ return (inavidation_cause == RS_INVAL_INACTIVE_TIMEOUT);
+ }
+
+ return false;
+}
+
/*
* Flush all replication slots to disk.
*
@@ -1835,6 +2003,7 @@ void
CheckPointReplicationSlots(bool is_shutdown)
{
int i;
+ bool invalidated = false;
elog(DEBUG1, "performing replication slot checkpoint");
@@ -1858,6 +2027,13 @@ CheckPointReplicationSlots(bool is_shutdown)
/* save the slot to disk, locking is handled in SaveSlotToPath() */
sprintf(path, "pg_replslot/%s", NameStr(s->data.name));
+ /*
+ * Here's an opportunity to invalidate inactive replication slots
+ * based on timeout, so let's do it.
+ */
+ if (InvalidateInactiveReplicationSlot(s, false))
+ invalidated = true;
+
/*
* Slot's data is not flushed each time the confirmed_flush LSN is
* updated as that could lead to frequent writes. However, we decide
@@ -1884,6 +2060,13 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ /* If the slot has been invalidated, recalculate the resource limits */
+ if (invalidated)
+ {
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index dd6c1d5a7e..9ad3e55704 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -539,7 +539,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index bc40c454de..96eeb8b7d2 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index c12784cbec..4149ff1ffe 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2971,6 +2971,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index baecde2841..2e1ad2eaca 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 7b937d1a0c..f0ac324ce9 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *standby_slot_names;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -245,7 +248,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait);
extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, bool failover);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_timeout_invalidation);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(void);
extern void ReplicationSlotSave(void);
@@ -264,6 +268,8 @@ extern bool InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause
XLogSegNo oldestSegno,
Oid dboid,
TransactionId snapshotConflictHorizon);
+extern bool InvalidateInactiveReplicationSlot(ReplicationSlot *slot,
+ bool persist_state);
extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_lock);
extern int ReplicationSlotIndex(ReplicationSlot *slot);
extern bool ReplicationSlotName(int index, Name name);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 712924c2fa..0437ab5c46 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -52,6 +52,7 @@ tests += {
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
't/043_wal_replay_wait.pl',
+ 't/044_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/044_invalidate_slots.pl b/src/test/recovery/t/044_invalidate_slots.pl
new file mode 100644
index 0000000000..6f1cccea55
--- /dev/null
+++ b/src/test/recovery/t/044_invalidate_slots.pl
@@ -0,0 +1,274 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to replication_slot_inactive_timeout. Also,
+# check the logical failover slot synced on to the standby doesn't invalidate
+# the slot on its own, but gets the invalidated state from the remote slot on
+# the primary.
+
+# Initialize primary node
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid checkpoint during the test, otherwise, the test can get unpredictable
+$primary->append_conf(
+ 'postgresql.conf', q{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create a standby linking to the primary using the replication slot
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr_1 = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb1_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot');
+]);
+
+$standby1->start;
+
+my $standby1_logstart = -s $standby1->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Synchronize the primary server slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced'.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot has synced as true on standby');
+
+my $logstart = -s $primary->logfile;
+
+# Set timeout so that the next checkpoint will invalidate the inactive
+# replication slot.
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$primary->reload;
+
+# Wait for the logical failover slot to become inactive on the primary. Note
+# that nobody has acquired that slot yet, so due to
+# replication_slot_inactive_timeout setting above it must get invalidated.
+wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart);
+
+# Set timeout on the standby also to check the synced slots don't get
+# invalidated due to timeout on the standby.
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$standby1->reload;
+
+# Now, sync the logical failover slot from the remote slot on the primary.
+# Note that the remote slot has already been invalidated due to inactive
+# timeout. Now, the standby must also see it as invalidated.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby";
+
+# Synced slot mustn't get invalidated on the standby, it must sync invalidation
+# from the primary. So, we must not see the slot's invalidation message in server
+# log.
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned slot has not been invalidated on the standby');
+
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart);
+
+# Testcase end: Invalidate streaming standby's slot as well as logical failover
+# slot on primary due to replication_slot_inactive_timeout. Also, check the
+# logical failover slot synced on to the standby doesn't invalidate the slot on
+# its own, but gets the invalidated state from the remote slot on the primary.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+
+my $publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber node
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$subscriber->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart);
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
+
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $name = $node->name;
+
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for replication slot to become inactive";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of replication slot $slot_name to be updated on node $name";
+
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive replication slot $slot_name to be invalidated on node $name";
+
+ # Check that the invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $primary->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot_name', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot_name"/,
+ "detected error upon trying to acquire invalidated slot $slot_name on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot_name";
+}
+
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot_name\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot_name invalidation has been logged");
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-05 07:43 ` Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bertrand Drouvot @ 2024-04-05 07:43 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On Fri, Apr 05, 2024 at 11:21:43AM +0530, Bharath Rupireddy wrote:
> On Wed, Apr 3, 2024 at 9:57 PM Bertrand Drouvot
> <[email protected]> wrote:
> Please find the attached v36 patch.
Thanks!
A few comments:
1 ===
+ <para>
+ The timeout is measured from the time since the slot has become
+ inactive (known from its
+ <structfield>inactive_since</structfield> value) until it gets
+ used (i.e., its <structfield>active</structfield> is set to true).
+ </para>
That's right except when it's invalidated during the checkpoint (as the slot
is not acquired in CheckPointReplicationSlots()).
So, what about adding: "or a checkpoint occurs"? That would also explain that
the invalidation could occur during checkpoint.
2 ===
+ /* If the slot has been invalidated, recalculate the resource limits */
+ if (invalidated)
+ {
/If the slot/If a slot/?
3 ===
+ * NB - this function also runs as part of checkpoint, so avoid raising errors
s/NB - this/NB: This function/? (that looks more consistent with other comments
in the code)
4 ===
+ * Note that having a new function for RS_INVAL_INACTIVE_TIMEOUT cause instead
I understand it's "the RS_INVAL_INACTIVE_TIMEOUT cause" but reading "cause instead"
looks weird to me. Maybe it would make sense to reword this a bit.
5 ===
+ * considered not active as they don't actually perform logical decoding.
Not sure that's 100% accurate as we switched in fast forward logical
in 2ec005b4e2.
"as they perform only fast forward logical decoding (or not at all)", maybe?
6 ===
+ if (RecoveryInProgress() && slot->data.synced)
+ return false;
+
+ if (replication_slot_inactive_timeout == 0)
+ return false;
What about just using one if? It's more a matter of taste but it also probably
reduces the object file size a bit for non optimized build.
7 ===
+ /*
+ * Do not invalidate the slots which are currently being synced from
+ * the primary to the standby.
+ */
+ if (RecoveryInProgress() && slot->data.synced)
+ return false;
I think we don't need this check as the exact same one is done just before.
8 ===
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
Wouldn't be better to wait for the replication_slot_inactive_timeout time before
instead of triggering all those checkpoints? (it could be passed as an extra arg
to wait_for_slot_invalidation()).
9 ===
# Synced slot mustn't get invalidated on the standby, it must sync invalidation
# from the primary. So, we must not see the slot's invalidation message in server
# log.
ok( !$standby1->log_contains(
"invalidating obsolete replication slot \"lsub1_sync_slot\"",
$standby1_logstart),
'check that syned slot has not been invalidated on the standby');
Would that make sense to trigger a checkpoint on the standby before this test?
I mean I think that without a checkpoint on the standby we should not see the
invalidation in the log anyway.
Regards,
--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
@ 2024-04-06 06:25 ` Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-04-06 06:25 UTC (permalink / raw)
To: Bertrand Drouvot <[email protected]>; +Cc: shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Apr 5, 2024 at 1:14 PM Bertrand Drouvot
<[email protected]> wrote:
>
> > Please find the attached v36 patch.
>
> A few comments:
>
> 1 ===
>
> + <para>
> + The timeout is measured from the time since the slot has become
> + inactive (known from its
> + <structfield>inactive_since</structfield> value) until it gets
> + used (i.e., its <structfield>active</structfield> is set to true).
> + </para>
>
> That's right except when it's invalidated during the checkpoint (as the slot
> is not acquired in CheckPointReplicationSlots()).
>
> So, what about adding: "or a checkpoint occurs"? That would also explain that
> the invalidation could occur during checkpoint.
Reworded.
> 2 ===
>
> + /* If the slot has been invalidated, recalculate the resource limits */
> + if (invalidated)
> + {
>
> /If the slot/If a slot/?
Modified it to be like elsewhere.
> 3 ===
>
> + * NB - this function also runs as part of checkpoint, so avoid raising errors
>
> s/NB - this/NB: This function/? (that looks more consistent with other comments
> in the code)
Done.
> 4 ===
>
> + * Note that having a new function for RS_INVAL_INACTIVE_TIMEOUT cause instead
>
> I understand it's "the RS_INVAL_INACTIVE_TIMEOUT cause" but reading "cause instead"
> looks weird to me. Maybe it would make sense to reword this a bit.
Reworded.
> 5 ===
>
> + * considered not active as they don't actually perform logical decoding.
>
> Not sure that's 100% accurate as we switched in fast forward logical
> in 2ec005b4e2.
>
> "as they perform only fast forward logical decoding (or not at all)", maybe?
Changed it to "as they don't perform logical decoding to produce the
changes". In fast_forward mode no changes are produced.
> 6 ===
>
> + if (RecoveryInProgress() && slot->data.synced)
> + return false;
> +
> + if (replication_slot_inactive_timeout == 0)
> + return false;
>
> What about just using one if? It's more a matter of taste but it also probably
> reduces the object file size a bit for non optimized build.
Changed.
> 7 ===
>
> + /*
> + * Do not invalidate the slots which are currently being synced from
> + * the primary to the standby.
> + */
> + if (RecoveryInProgress() && slot->data.synced)
> + return false;
>
> I think we don't need this check as the exact same one is done just before.
Right. Removed.
> 8 ===
>
> +sub check_for_slot_invalidation_in_server_log
> +{
> + my ($node, $slot_name, $offset) = @_;
> + my $invalidated = 0;
> +
> + for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
> + {
> + $node->safe_psql('postgres', "CHECKPOINT");
>
> Wouldn't be better to wait for the replication_slot_inactive_timeout time before
> instead of triggering all those checkpoints? (it could be passed as an extra arg
> to wait_for_slot_invalidation()).
Done.
> 9 ===
>
> # Synced slot mustn't get invalidated on the standby, it must sync invalidation
> # from the primary. So, we must not see the slot's invalidation message in server
> # log.
> ok( !$standby1->log_contains(
> "invalidating obsolete replication slot \"lsub1_sync_slot\"",
> $standby1_logstart),
> 'check that syned slot has not been invalidated on the standby');
>
> Would that make sense to trigger a checkpoint on the standby before this test?
> I mean I think that without a checkpoint on the standby we should not see the
> invalidation in the log anyway.
Done.
Please find the attached v37 patch for further review.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v37-0001-Add-inactive_timeout-based-replication-slot-inva.patch (32.4K, ../../CALj2ACW8SYiRL8vGAhez=FdrzxO0W5pVFvN8GF7yKCO8_c=QdQ@mail.gmail.com/2-v37-0001-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From 648d6f55d4b1593d0f09ab3ee8dcc91d57bf4961 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Sat, 6 Apr 2024 06:22:28 +0000
Subject: [PATCH v37] Add inactive_timeout based replication slot invalidation.
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set a timeout of say 1
or 2 or 3 days at slot level, after which the inactive slots get
invalidated.
To achieve the above, postgres introduces a GUC allowing users
set inactive timeout. The replication slots that are inactive
for longer than specified amount of time get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Note that this new invalidation mechanism won't kick-in for the
slots that are currently being synced from the primary to the
standby.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
---
doc/src/sgml/config.sgml | 33 ++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 11 +-
src/backend/replication/slot.c | 211 ++++++++++++-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 8 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/044_invalidate_slots.pl | 283 ++++++++++++++++++
13 files changed, 553 insertions(+), 24 deletions(-)
create mode 100644 src/test/recovery/t/044_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 624518e0b0..42f7b15aa7 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4547,6 +4547,39 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during a checkpoint. The time since the slot has become
+ inactive is known from its
+ <structfield>inactive_since</structfield> value using which the
+ timeout is measured.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 7ed617170f..063638beda 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2580,6 +2580,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for the duration specified by
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index d18e2c7342..e92f559539 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -362,7 +362,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -575,6 +575,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
" name slot \"%s\" already exists on the standby",
remote_slot->name));
+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to avoid slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
+
/*
* The slot has been synchronized before.
*
@@ -591,7 +598,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 3bddaae022..4a43b29d89 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -158,6 +160,7 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyInactiveSlot(ReplicationSlot *slot);
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
@@ -535,9 +538,14 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * If check_for_timeout_invalidation is true, the slot is checked for
+ * invalidation based on replication_slot_inactive_timeout GUC, and an error is
+ * raised after making the slot ours.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_timeout_invalidation)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +623,34 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * Check if the given slot can be invalidated based on its inactive
+ * timeout. If yes, persist the invalidated state to disk and then error
+ * out. We do this only after making the slot ours to avoid anyone else
+ * acquiring it while we check for its invalidation.
+ */
+ if (check_for_timeout_invalidation)
+ {
+ /* The slot is ours by now */
+ Assert(s->active_pid == MyProcPid);
+
+ if (InvalidateInactiveReplicationSlot(s, true))
+ {
+ /*
+ * If the slot has been invalidated, recalculate the resource
+ * limits.
+ */
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(MyReplicationSlot->data.name)),
+ errdetail("This slot has been invalidated because it was inactive for more than replication_slot_inactive_timeout.")));
+ }
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -781,7 +817,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -804,7 +840,7 @@ ReplicationSlotAlter(const char *name, bool failover)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -980,6 +1016,20 @@ ReplicationSlotDropPtr(ReplicationSlot *slot)
LWLockRelease(ReplicationSlotAllocationLock);
}
+/*
+ * Helper for ReplicationSlotSave
+ */
+static inline void
+SaveGivenReplicationSlot(ReplicationSlot *slot, int elevel)
+{
+ char path[MAXPGPATH];
+
+ Assert(slot != NULL);
+
+ sprintf(path, "pg_replslot/%s", NameStr(slot->data.name));
+ SaveSlotToPath(slot, path, elevel);
+}
+
/*
* Serialize the currently acquired slot's state from memory to disk, thereby
* guaranteeing the current state will survive a crash.
@@ -987,12 +1037,21 @@ ReplicationSlotDropPtr(ReplicationSlot *slot)
void
ReplicationSlotSave(void)
{
- char path[MAXPGPATH];
+ SaveGivenReplicationSlot(MyReplicationSlot, ERROR);
+}
- Assert(MyReplicationSlot != NULL);
+/*
+ * Helper for ReplicationSlotMarkDirty
+ */
+static inline void
+MarkGivenReplicationSlotDirty(ReplicationSlot *slot)
+{
+ Assert(slot != NULL);
- sprintf(path, "pg_replslot/%s", NameStr(MyReplicationSlot->data.name));
- SaveSlotToPath(MyReplicationSlot, path, ERROR);
+ SpinLockAcquire(&slot->mutex);
+ slot->just_dirtied = true;
+ slot->dirty = true;
+ SpinLockRelease(&slot->mutex);
}
/*
@@ -1005,14 +1064,7 @@ ReplicationSlotSave(void)
void
ReplicationSlotMarkDirty(void)
{
- ReplicationSlot *slot = MyReplicationSlot;
-
- Assert(MyReplicationSlot != NULL);
-
- SpinLockAcquire(&slot->mutex);
- MyReplicationSlot->just_dirtied = true;
- MyReplicationSlot->dirty = true;
- SpinLockRelease(&slot->mutex);
+ MarkGivenReplicationSlotDirty(MyReplicationSlot);
}
/*
@@ -1506,6 +1558,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires wal_level >= logical on the primary server."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ appendStringInfoString(&err_detail, _("The slot has been inactive for more than replication_slot_inactive_timeout."));
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1550,6 +1605,12 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ /*
+ * Use InvalidateInactiveReplicationSlot for inactive timeout based
+ * invalidation.
+ */
+ Assert(cause != RS_INVAL_INACTIVE_TIMEOUT);
+
for (;;)
{
XLogRecPtr restart_lsn;
@@ -1619,6 +1680,10 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ /* not reachable */
+ Assert(false);
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1787,6 +1852,12 @@ InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause cause,
Assert(cause != RS_INVAL_WAL_REMOVED || oldestSegno > 0);
Assert(cause != RS_INVAL_NONE);
+ /*
+ * Use InvalidateInactiveReplicationSlot for inactive timeout based
+ * invalidation.
+ */
+ Assert(cause != RS_INVAL_INACTIVE_TIMEOUT);
+
if (max_replication_slots == 0)
return invalidated;
@@ -1823,6 +1894,97 @@ restart:
return invalidated;
}
+/*
+ * Invalidate given slot based on replication_slot_inactive_timeout GUC.
+ *
+ * Returns true if the slot has got invalidated.
+ *
+ * Whether the given slot needs to be invalidated depends on the cause:
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout occurs
+ *
+ * NB: This function also runs as part of checkpoint, so avoid raising errors
+ * if possible.
+ *
+ * Note that having a new function for inactive timeout invalidation mechanism
+ * instead of using InvalidateObsoleteReplicationSlots provides flexibility in
+ * calling it at slot-level opportunistically, and choosing whether or not to
+ * persist the slot info to disk.
+ */
+bool
+InvalidateInactiveReplicationSlot(ReplicationSlot *slot, bool persist_state)
+{
+ if (!InvalidatePossiblyInactiveSlot(slot))
+ return false;
+
+ /* Make sure the invalidated state persists across server restart */
+ MarkGivenReplicationSlotDirty(slot);
+
+ if (persist_state)
+ SaveGivenReplicationSlot(slot, ERROR);
+
+ ReportSlotInvalidation(RS_INVAL_INACTIVE_TIMEOUT, false, 0,
+ slot->data.name, InvalidXLogRecPtr,
+ InvalidXLogRecPtr, InvalidTransactionId);
+
+ return true;
+}
+
+/*
+ * Helper for InvalidateInactiveReplicationSlot
+ */
+static bool
+InvalidatePossiblyInactiveSlot(ReplicationSlot *slot)
+{
+ ReplicationSlotInvalidationCause inavidation_cause = RS_INVAL_NONE;
+
+ /*
+ * Quick exit if inactive timeout invalidation mechanism is disabled or
+ * the slot on standby is currently being synced from the primary.
+ *
+ * Note that we don't invalidate synced slots because, they are typically
+ * considered not active as they don't perform logical decoding to produce
+ * the changes.
+ */
+ if (replication_slot_inactive_timeout == 0 ||
+ (RecoveryInProgress() && slot->data.synced))
+ return false;
+
+ if (slot->inactive_since > 0)
+ {
+ TimestampTz now;
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+
+ SpinLockAcquire(&slot->mutex);
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC. We do this with the spinlock
+ * held to avoid race conditions -- for example the inactive_since
+ * could change, or the slot could be dropped.
+ */
+ if (TimestampDifferenceExceeds(slot->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ inavidation_cause = RS_INVAL_INACTIVE_TIMEOUT;
+ slot->data.invalidated = RS_INVAL_INACTIVE_TIMEOUT;
+ }
+
+ SpinLockRelease(&slot->mutex);
+ LWLockRelease(ReplicationSlotControlLock);
+
+ return (inavidation_cause == RS_INVAL_INACTIVE_TIMEOUT);
+ }
+
+ return false;
+}
+
/*
* Flush all replication slots to disk.
*
@@ -1835,6 +1997,7 @@ void
CheckPointReplicationSlots(bool is_shutdown)
{
int i;
+ bool invalidated = false;
elog(DEBUG1, "performing replication slot checkpoint");
@@ -1858,6 +2021,13 @@ CheckPointReplicationSlots(bool is_shutdown)
/* save the slot to disk, locking is handled in SaveSlotToPath() */
sprintf(path, "pg_replslot/%s", NameStr(s->data.name));
+ /*
+ * Here's an opportunity to invalidate inactive replication slots
+ * based on timeout, so let's do it.
+ */
+ if (InvalidateInactiveReplicationSlot(s, false))
+ invalidated = true;
+
/*
* Slot's data is not flushed each time the confirmed_flush LSN is
* updated as that could lead to frequent writes. However, we decide
@@ -1884,6 +2054,15 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ /*
+ * If any slots have been invalidated, recalculate the resource limits.
+ */
+ if (invalidated)
+ {
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index dd6c1d5a7e..9ad3e55704 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -539,7 +539,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index bc40c454de..96eeb8b7d2 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index c12784cbec..4149ff1ffe 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2971,6 +2971,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index baecde2841..2e1ad2eaca 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 7b937d1a0c..f0ac324ce9 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *standby_slot_names;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -245,7 +248,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait);
extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, bool failover);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_timeout_invalidation);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(void);
extern void ReplicationSlotSave(void);
@@ -264,6 +268,8 @@ extern bool InvalidateObsoleteReplicationSlots(ReplicationSlotInvalidationCause
XLogSegNo oldestSegno,
Oid dboid,
TransactionId snapshotConflictHorizon);
+extern bool InvalidateInactiveReplicationSlot(ReplicationSlot *slot,
+ bool persist_state);
extern ReplicationSlot *SearchNamedReplicationSlot(const char *name, bool need_lock);
extern int ReplicationSlotIndex(ReplicationSlot *slot);
extern bool ReplicationSlotName(int index, Name name);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 712924c2fa..0437ab5c46 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -52,6 +52,7 @@ tests += {
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
't/043_wal_replay_wait.pl',
+ 't/044_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/044_invalidate_slots.pl b/src/test/recovery/t/044_invalidate_slots.pl
new file mode 100644
index 0000000000..1e30ea05ef
--- /dev/null
+++ b/src/test/recovery/t/044_invalidate_slots.pl
@@ -0,0 +1,283 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to replication_slot_inactive_timeout. Also,
+# check the logical failover slot synced on to the standby doesn't invalidate
+# the slot on its own, but gets the invalidated state from the remote slot on
+# the primary.
+
+# Initialize primary node
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid checkpoint during the test, otherwise, the test can get unpredictable
+$primary->append_conf(
+ 'postgresql.conf', q{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create a standby linking to the primary using the replication slot
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr_1 = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb1_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot');
+]);
+
+$standby1->start;
+
+my $standby1_logstart = -s $standby1->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Synchronize the primary server slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced'.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot has synced as true on standby');
+
+my $logstart = -s $primary->logfile;
+my $inactive_timeout = 2;
+
+# Set timeout so that the next checkpoint will invalidate the inactive
+# replication slot.
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for the logical failover slot to become inactive on the primary. Note
+# that nobody has acquired that slot yet, so due to
+# replication_slot_inactive_timeout setting above it must get invalidated.
+wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
+ $inactive_timeout);
+
+# Set timeout on the standby also to check the synced slots don't get
+# invalidated due to timeout on the standby.
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$standby1->reload;
+
+# Now, sync the logical failover slot from the remote slot on the primary.
+# Note that the remote slot has already been invalidated due to inactive
+# timeout. Now, the standby must also see it as invalidated.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby";
+
+# Synced slot mustn't get invalidated on the standby even after a checkpoint,
+# it must sync invalidation from the primary. So, we must not see the slot's
+# invalidation message in server log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned slot has not been invalidated on the standby');
+
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate streaming standby's slot as well as logical failover
+# slot on primary due to replication_slot_inactive_timeout. Also, check the
+# logical failover slot synced on to the standby doesn't invalidate the slot on
+# its own, but gets the invalidated state from the remote slot on the primary.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+
+my $publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber node
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$subscriber->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
+
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for replication slot to become inactive";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of replication slot $slot_name to be updated on node $name";
+
+ # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
+ # for the slot to get invalidated.
+ sleep($inactive_timeout);
+
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive replication slot $slot_name to be invalidated on node $name";
+
+ # Check that the invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $primary->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot_name', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot_name"/,
+ "detected error upon trying to acquire invalidated slot $slot_name on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot_name";
+}
+
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot_name\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot_name invalidation has been logged");
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-06 06:48 ` Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Amit Kapila @ 2024-04-06 06:48 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sat, Apr 6, 2024 at 11:55 AM Bharath Rupireddy
<[email protected]> wrote:
>
Why the handling w.r.t active_pid in InvalidatePossiblyInactiveSlot()
is not similar to InvalidatePossiblyObsoleteSlot(). Won't we need to
ensure that there is no other active slot user? Is it sufficient to
check inactive_since for the same? If so, we need some comments to
explain the same.
Can we avoid introducing the new functions like
SaveGivenReplicationSlot() and MarkGivenReplicationSlotDirty(), if we
do the required work in the caller?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
@ 2024-04-06 11:40 ` Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-04-06 11:40 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sat, Apr 6, 2024 at 12:18 PM Amit Kapila <[email protected]> wrote:
>
> Why the handling w.r.t active_pid in InvalidatePossiblyInactiveSlot()
> is not similar to InvalidatePossiblyObsoleteSlot(). Won't we need to
> ensure that there is no other active slot user? Is it sufficient to
> check inactive_since for the same? If so, we need some comments to
> explain the same.
I removed the separate functions and with minimal changes, I've now
placed the RS_INVAL_INACTIVE_TIMEOUT logic into
InvalidatePossiblyObsoleteSlot and use that even in
CheckPointReplicationSlots.
> Can we avoid introducing the new functions like
> SaveGivenReplicationSlot() and MarkGivenReplicationSlotDirty(), if we
> do the required work in the caller?
Hm. Removed them now.
Please see the attached v38 patch.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v38-0001-Add-inactive_timeout-based-replication-slot-inva.patch (31.5K, ../../CALj2ACWhTY9mRc7sOpuao=GGidRjCVXc0SNtJ_DNman4nh5CcQ@mail.gmail.com/2-v38-0001-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From a2c66c711da7118eae48446c87e5595f85c8e45a Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Sat, 6 Apr 2024 11:13:02 +0000
Subject: [PATCH v38] Add inactive_timeout based replication slot invalidation.
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set a timeout of say 1
or 2 or 3 days at slot level, after which the inactive slots get
invalidated.
To achieve the above, postgres introduces a GUC allowing users
set inactive timeout. The replication slots that are inactive
for longer than specified amount of time get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Note that this new invalidation mechanism won't kick-in for the
slots that are currently being synced from the primary to the
standby.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
---
doc/src/sgml/config.sgml | 33 ++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 11 +-
src/backend/replication/slot.c | 174 ++++++++++-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 6 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/044_invalidate_slots.pl | 283 ++++++++++++++++++
13 files changed, 523 insertions(+), 15 deletions(-)
create mode 100644 src/test/recovery/t/044_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 624518e0b0..42f7b15aa7 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4547,6 +4547,39 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during a checkpoint. The time since the slot has become
+ inactive is known from its
+ <structfield>inactive_since</structfield> value using which the
+ timeout is measured.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 7ed617170f..063638beda 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2580,6 +2580,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for the duration specified by
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index d18e2c7342..e92f559539 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -362,7 +362,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -575,6 +575,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
" name slot \"%s\" already exists on the standby",
remote_slot->name));
+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to avoid slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
+
/*
* The slot has been synchronized before.
*
@@ -591,7 +598,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 3bddaae022..767d818706 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -159,6 +161,14 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
+ ReplicationSlot *s,
+ XLogRecPtr oldestLSN,
+ Oid dboid,
+ TransactionId snapshotConflictHorizon,
+ bool need_lock,
+ bool *invalidated);
+
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
static void CreateSlotOnDisk(ReplicationSlot *slot);
@@ -535,12 +545,18 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * If check_for_timeout_invalidation is true, the slot is checked for
+ * invalidation based on replication_slot_inactive_timeout GUC, and an error is
+ * raised after making the slot ours.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_timeout_invalidation)
{
ReplicationSlot *s;
int active_pid;
+ bool released_lock = false;
Assert(name != NULL);
@@ -583,7 +599,60 @@ retry:
}
else
active_pid = MyProcPid;
- LWLockRelease(ReplicationSlotControlLock);
+
+ /*
+ * When the slot is not active under other process, check if the given
+ * slot is to be invalidated based on inactive timeout before we acquire
+ * it.
+ */
+ if (active_pid == MyProcPid &&
+ check_for_timeout_invalidation)
+ {
+ bool invalidated = false;
+
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_INACTIVE_TIMEOUT,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ false,
+ &invalidated);
+
+ /*
+ * If the slot has been invalidated, recalculate the resource limits.
+ */
+ if (invalidated)
+ {
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
+
+ /*
+ * If the slot has been invalidated now or previously, error out as
+ * there's no point in acquiring the slot.
+ *
+ * XXX: We might need to check for all invalidations and error out
+ * here.
+ */
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ /*
+ * Release the lock if it's not yet and make this slot ours to
+ * keep the cleanup path on error is happy.
+ */
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
+ MyReplicationSlot = s;
+
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive for more than replication_slot_inactive_timeout.")));
+ }
+ }
+
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
/*
* If we found the slot but it's already active in another process, we
@@ -781,7 +850,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -804,7 +873,7 @@ ReplicationSlotAlter(const char *name, bool failover)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1506,6 +1575,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires wal_level >= logical on the primary server."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ appendStringInfoString(&err_detail, _("The slot has been inactive for more than replication_slot_inactive_timeout."));
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1540,7 +1612,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReplicationSlot *s,
XLogRecPtr oldestLSN,
Oid dboid, TransactionId snapshotConflictHorizon,
- bool *invalidated)
+ bool need_lock, bool *invalidated)
{
int last_signaled_pid = 0;
bool released_lock = false;
@@ -1550,12 +1622,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ if (need_lock)
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
for (;;)
{
XLogRecPtr restart_lsn;
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1566,6 +1642,18 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced)))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1619,6 +1707,41 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ {
+ /*
+ * Quick exit if inactive timeout invalidation
+ * mechanism is disabled or slot is currently being
+ * used or the slot on standby is currently being
+ * synced from the primary.
+ *
+ * Note that we don't invalidate synced slots because,
+ * they are typically considered not active as they
+ * don't perform logical decoding to produce the
+ * changes.
+ */
+ if (replication_slot_inactive_timeout == 0 ||
+ s->inactive_since == 0 ||
+ (RecoveryInProgress() && s->data.synced))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+
+ /*
+ * Invalidation due to inactive timeout implies no
+ * one using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1756,6 +1879,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
}
}
+ /*
+ * Release the lock if we have acquired it at the beginning, and have not
+ * released it yet.
+ */
+ if (need_lock && !released_lock)
+ {
+ LWLockRelease(ReplicationSlotControlLock);
+ released_lock = true;
+ }
+
Assert(released_lock == !LWLockHeldByMe(ReplicationSlotControlLock));
return released_lock;
@@ -1772,6 +1905,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1803,7 +1937,7 @@ restart:
if (InvalidatePossiblyObsoleteSlot(cause, s, oldestLSN, dboid,
snapshotConflictHorizon,
- &invalidated))
+ false, &invalidated))
{
/* if the lock was released, start from scratch */
goto restart;
@@ -1835,6 +1969,7 @@ void
CheckPointReplicationSlots(bool is_shutdown)
{
int i;
+ bool invalidated = false;
elog(DEBUG1, "performing replication slot checkpoint");
@@ -1851,10 +1986,26 @@ CheckPointReplicationSlots(bool is_shutdown)
{
ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
char path[MAXPGPATH];
+ bool released_lock PG_USED_FOR_ASSERTS_ONLY = false;
if (!s->in_use)
continue;
+ /*
+ * Here's an opportunity to invalidate inactive replication slots
+ * based on timeout, so let's do it.
+ */
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_INACTIVE_TIMEOUT, s,
+ 0, InvalidOid,
+ InvalidTransactionId,
+ true, &invalidated);
+
+ /*
+ * InvalidatePossiblyObsoleteSlot must have released the lock as we
+ * have told it explicitly acquire the lock.
+ */
+ Assert(released_lock);
+
/* save the slot to disk, locking is handled in SaveSlotToPath() */
sprintf(path, "pg_replslot/%s", NameStr(s->data.name));
@@ -1884,6 +2035,15 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ /*
+ * If any slots have been invalidated, recalculate the resource limits.
+ */
+ if (invalidated)
+ {
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index dd6c1d5a7e..9ad3e55704 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -539,7 +539,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index bc40c454de..96eeb8b7d2 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index c12784cbec..4149ff1ffe 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2971,6 +2971,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index baecde2841..2e1ad2eaca 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 7b937d1a0c..3f3bdf7441 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *standby_slot_names;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -245,7 +248,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait);
extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, bool failover);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_timeout_invalidation);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(void);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 712924c2fa..0437ab5c46 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -52,6 +52,7 @@ tests += {
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
't/043_wal_replay_wait.pl',
+ 't/044_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/044_invalidate_slots.pl b/src/test/recovery/t/044_invalidate_slots.pl
new file mode 100644
index 0000000000..7796d87963
--- /dev/null
+++ b/src/test/recovery/t/044_invalidate_slots.pl
@@ -0,0 +1,283 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to replication_slot_inactive_timeout. Also,
+# check the logical failover slot synced on to the standby doesn't invalidate
+# the slot on its own, but gets the invalidated state from the remote slot on
+# the primary.
+
+# Initialize primary node
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid checkpoint during the test, otherwise, the test can get unpredictable
+$primary->append_conf(
+ 'postgresql.conf', q{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create a standby linking to the primary using the replication slot
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr_1 = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb1_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot');
+]);
+
+$standby1->start;
+
+my $standby1_logstart = -s $standby1->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Synchronize the primary server slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced'.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot has synced as true on standby');
+
+my $logstart = -s $primary->logfile;
+my $inactive_timeout = 2;
+
+# Set timeout so that the next checkpoint will invalidate the inactive
+# replication slot.
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for the logical failover slot to become inactive on the primary. Note
+# that nobody has acquired that slot yet, so due to
+# replication_slot_inactive_timeout setting above it must get invalidated.
+wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
+ $inactive_timeout);
+
+# Set timeout on the standby also to check the synced slots don't get
+# invalidated due to timeout on the standby.
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$standby1->reload;
+
+# Now, sync the logical failover slot from the remote slot on the primary.
+# Note that the remote slot has already been invalidated due to inactive
+# timeout. Now, the standby must also see it as invalidated.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby";
+
+# Synced slot mustn't get invalidated on the standby even after a checkpoint,
+# it must sync invalidation from the primary. So, we must not see the slot's
+# invalidation message in server log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned slot has not been invalidated on the standby');
+
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate streaming standby's slot as well as logical failover
+# slot on primary due to replication_slot_inactive_timeout. Also, check the
+# logical failover slot synced on to the standby doesn't invalidate the slot on
+# its own, but gets the invalidated state from the remote slot on the primary.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+
+my $publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber node
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$subscriber->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
+
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for replication slot to become inactive";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of replication slot $slot_name to be updated on node $name";
+
+ # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
+ # for the slot to get invalidated.
+ sleep($inactive_timeout);
+
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive replication slot $slot_name to be invalidated on node $name";
+
+ # Check that the invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot_name', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot_name"/,
+ "detected error upon trying to acquire invalidated slot $slot_name on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot_name";
+}
+
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot_name\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot_name invalidation has been logged");
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-13 04:06 ` Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-04-13 04:06 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sat, Apr 6, 2024 at 5:10 PM Bharath Rupireddy
<[email protected]> wrote:
>
> Please see the attached v38 patch.
Hi, thanks everyone for reviewing the design and patches so far. Here
I'm with the v39 patches implementing inactive timeout based (0001)
and XID age based (0002) invalidation mechanisms.
I'm quoting the hackers who are okay with inactive timeout based
invalidation mechanism:
Bertrand Drouvot -
https://www.postgresql.org/message-id/ZgL0N%2BxVJNkyqsKL%40ip-10-97-1-34.eu-west-3.compute.internal
and https://www.postgresql.org/message-id/ZgPHDAlM79iLtGIH%40ip-10-97-1-34.eu-west-3.compute.internal
Amit Kapila - https://www.postgresql.org/message-id/CAA4eK1L3awyzWMuymLJUm8SoFEQe%3DDa9KUwCcAfC31RNJ1xdJA%40mail.g...
Nathan Bossart -
https://www.postgresql.org/message-id/20240325195443.GA2923888%40nathanxps13
Robert Haas - https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.g...
I'm quoting the hackers who are okay with XID age based invalidation mechanism:
Nathan Bossart -
https://www.postgresql.org/message-id/20240326150918.GB3181099%40nathanxps13
and https://www.postgresql.org/message-id/20240327150557.GA3994937%40nathanxps13
Alvaro Herrera -
https://www.postgresql.org/message-id/202403261539.xcjfle7sksz7%40alvherre.pgsql
Bertrand Drouvot -
https://www.postgresql.org/message-id/ZgPHDAlM79iLtGIH%40ip-10-97-1-34.eu-west-3.compute.internal
Amit Kapila - https://www.postgresql.org/message-id/CAA4eK1L3awyzWMuymLJUm8SoFEQe%3DDa9KUwCcAfC31RNJ1xdJA%40mail.g...
There was a point raised by Robert
https://www.postgresql.org/message-id/CA%2BTgmoaRECcnyqxAxUhP5dk2S4HX%3DpGh-p-PkA3uc%2BjG_9hiMw%40ma...
for XID age based invalidation. An issue related to
vacuum_defer_cleanup_age
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=be504a3e974d75be6f95c8f9b73671260...
led to the removal of the GUC
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=1118cd37eb61e6a2428f457a8b2026a7b....
The same issue may not happen for the XID age based invaliation. This
is because the XID age is not calculated using FullTransactionId but
using TransactionId as the slot's xmin and catalog_xmin are tracked as
TransactionId.
There was a point raised by Amit
https://www.postgresql.org/message-id/CAA4eK1K8wqLsMw6j0hE_SFoWAeo3Kw8UNnMfhsWaYDF1GWYQ%2Bg%40mail.g...
on when to do the XID age based invalidation - whether in checkpointer
or when vacuum is being run or whenever ComputeXIDHorizons gets called
or in autovacuum process. For now, I've chosen the design to do these
new invalidation checks in two places - 1) whenever the slot is
acquired and the slot acquisition errors out if invalidated, 2) during
checkpoint. However, I'm open to suggestions on this.
I've also verified the case whether the replication_slot_xid_age
setting can help in case of server inching towards the XID wraparound.
I've created a primary and streaming standby setup with
hot_standby_feedback set to on (so that the slot gets an xmin). Then,
I've set replication_slot_xid_age to 2 billion on the primary, and
used xid_wraparound extension to reach XID wraparound on the primary.
Once I start receiving the WARNINGs about VACUUM, I did a checkpoint
after which the slot got invalidated enabling my VACUUM to freeze XIDs
saving my database from XID wraparound problem.
Thanks a lot Masahiko Sawada for an offlist chat about the XID age
calculation logic.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/x-patch] v39-0001-Add-inactive_timeout-based-replication-slot-inva.patch (33.6K, ../../CALj2ACVY+Fd5vC0VjW=5VDK9mmt-Y+PDZxnBp8ngGAZc24Vv9g@mail.gmail.com/2-v39-0001-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From f3ba2562ba7d9c4f13e283740260025b8d1c9b0f Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Fri, 12 Apr 2024 14:52:35 +0000
Subject: [PATCH v39 1/2] Add inactive_timeout based replication slot
invalidation.
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set a timeout of say 1
or 2 or 3 days, after which the inactive slots get invalidated.
To achieve the above, postgres introduces a GUC allowing users
set inactive timeout. The replication slots that are inactive
for longer than specified amount of time get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Note that this new invalidation mechanism won't kick-in for the
slots that are currently being synced from the primary to the
standby, because such synced slots are typically considered not
active (for them to be later considered as inactive) as they don't
perform logical decoding to produce the changes.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/202403260841.5jcv7ihniccy%40alvherre.pgsql
---
doc/src/sgml/config.sgml | 33 ++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 11 +-
src/backend/replication/slot.c | 188 +++++++++++-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 6 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 286 ++++++++++++++++++
13 files changed, 535 insertions(+), 20 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d8e1282e12..a73677b98b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4547,6 +4547,39 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during a checkpoint. The time since the slot has become
+ inactive is known from its
+ <structfield>inactive_since</structfield> value using which the
+ timeout is measured.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 7ed617170f..063638beda 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2580,6 +2580,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for the duration specified by
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index bda0de52db..c47e56f78f 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -450,7 +450,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -653,6 +653,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
" name slot \"%s\" already exists on the standby",
remote_slot->name));
+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to avoid slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
+
/*
* The slot has been synchronized before.
*
@@ -669,7 +676,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index cebf44bb0f..7cfbc2dfff 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -159,6 +161,13 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
+ ReplicationSlot *s,
+ XLogRecPtr oldestLSN,
+ Oid dboid,
+ TransactionId snapshotConflictHorizon,
+ bool *invalidated);
+
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
static void CreateSlotOnDisk(ReplicationSlot *slot);
@@ -535,12 +544,17 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if check_for_invalidation is true and the slot gets
+ * invalidated now or has been invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation)
{
ReplicationSlot *s;
int active_pid;
+ bool released_lock = false;
Assert(name != NULL);
@@ -615,6 +629,57 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ /*
+ * Check if the acquired slot needs to be invalidated. And, error out if
+ * it gets invalidated now or has been invalidated previously, because
+ * there's no use in acquiring the invalidated slot.
+ *
+ * XXX: Currently we check for inactive_timeout invalidation here. We
+ * might need to check for other invalidations too.
+ */
+ if (check_for_invalidation)
+ {
+ bool invalidated = false;
+
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_INACTIVE_TIMEOUT,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ &invalidated);
+
+ /*
+ * If the slot has been invalidated, recalculate the resource limits.
+ */
+ if (invalidated)
+ {
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
+
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ /*
+ * Release the lock if it's not yet to keep the cleanup path on
+ * error happy.
+ */
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive since %s for more than replication_slot_inactive_timeout = %d seconds.",
+ timestamptz_to_str(s->inactive_since),
+ replication_slot_inactive_timeout)));
+ }
+ }
+
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -781,7 +846,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -804,7 +869,7 @@ ReplicationSlotAlter(const char *name, bool failover)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1476,7 +1541,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1506,6 +1572,13 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires wal_level >= logical on the primary server."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for more than replication_slot_inactive_timeout = %d seconds."),
+ timestamptz_to_str(inactive_since),
+ replication_slot_inactive_timeout);
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1549,6 +1622,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1556,6 +1630,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1566,6 +1641,18 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced)))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1619,6 +1706,39 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Quick exit if inactive timeout invalidation mechanism
+ * is disabled or slot is currently being used or the slot
+ * on standby is currently being synced from the primary.
+ *
+ * Note that we don't invalidate synced slots because,
+ * they are typically considered not active as they don't
+ * perform logical decoding to produce the changes.
+ */
+ if (replication_slot_inactive_timeout == 0 ||
+ s->inactive_since == 0 ||
+ (RecoveryInProgress() && s->data.synced))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1644,11 +1764,14 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so or if the slot is already ours,
+ * then mark it invalidated. Otherwise we'll signal the owning
+ * process, below, and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot != NULL &&
+ MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1703,7 +1826,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1749,7 +1873,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1772,6 +1897,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1824,7 +1950,7 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk and invalidate slots.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1835,6 +1961,7 @@ void
CheckPointReplicationSlots(bool is_shutdown)
{
int i;
+ bool invalidated = false;
elog(DEBUG1, "performing replication slot checkpoint");
@@ -1884,6 +2011,43 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ elog(DEBUG1, "performing replication slot invalidation");
+
+ /*
+ * Note that we will make another pass over replication slots for
+ * invalidations to keep the code simple. The assumption here is that the
+ * traversal over replication slots isn't that costly even with hundreds
+ * of replication slots. If it ever turns out that this assumption is
+ * wrong, we might have to put the invalidation check logic in the above
+ * loop, for that we might need to do the following:
+ *
+ * - Acqure ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated slot.
+ *
+ * XXX: Should we move inactive_timeout inavalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+
+ if (invalidated)
+ {
+ /*
+ * If any slots have been invalidated, recalculate the resource
+ * limits.
+ */
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index dd6c1d5a7e..9ad3e55704 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -539,7 +539,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index bc40c454de..96eeb8b7d2 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index c68fdc008b..79e7637ec9 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2982,6 +2982,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 2166ea4a87..819310b0a7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 7b937d1a0c..8727b7b58b 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *standby_slot_names;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -245,7 +248,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait);
extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, bool failover);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(void);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..b4c5ce2875 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -10,6 +10,7 @@ tests += {
'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
},
'tests': [
+ 't/050_invalidate_slots.pl',
't/001_stream_rep.pl',
't/002_archiving.pl',
't/003_recovery_targets.pl',
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..4663019c16
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,286 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to replication_slot_inactive_timeout. Also,
+# check the logical failover slot synced on to the standby doesn't invalidate
+# the slot on its own, but gets the invalidated state from the remote slot on
+# the primary.
+
+# Initialize primary node
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid checkpoint during the test, otherwise, the test can get unpredictable
+$primary->append_conf(
+ 'postgresql.conf', q{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create a standby linking to the primary using the replication slot
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr_1 = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb1_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+my $standby1_logstart = -s $standby1->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Synchronize the primary server slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced'.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot lsub1_sync_slot has synced as true on standby');
+
+my $logstart = -s $primary->logfile;
+my $inactive_timeout = 2;
+
+# Set timeout so that the next checkpoint will invalidate the inactive
+# replication slot.
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for the logical failover slot to become inactive on the primary. Note
+# that nobody has acquired that slot yet, so due to
+# replication_slot_inactive_timeout setting above it must get invalidated.
+wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
+ $inactive_timeout);
+
+# Set timeout on the standby also to check the synced slots don't get
+# invalidated due to timeout on the standby.
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$standby1->reload;
+
+# Now, sync the logical failover slot from the remote slot on the primary.
+# Note that the remote slot has already been invalidated due to inactive
+# timeout. Now, the standby must also see it as invalidated.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby";
+
+# Synced slot mustn't get invalidated on the standby even after a checkpoint,
+# it must sync invalidation from the primary. So, we must not see the slot's
+# invalidation message in server log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned slot lsub1_sync_slot has not been invalidated on the standby'
+);
+
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate streaming standby's slot as well as logical failover
+# slot on primary due to replication_slot_inactive_timeout. Also, check the
+# logical failover slot synced on to the standby doesn't invalidate the slot on
+# its own, but gets the invalidated state from the remote slot on the primary.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+
+my $publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber node
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
+
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for slot $slot_name to become inactive on node $name";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of slot $slot_name to be updated on node $name";
+
+ # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
+ # for the slot to get invalidated.
+ sleep($inactive_timeout);
+
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
+
+ # Check that the invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot_name', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot_name"/,
+ "detected error upon trying to acquire invalidated slot $slot_name on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot_name on node $name";
+}
+
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $name = $node->name;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot_name\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot_name invalidation has been logged on node $name"
+ );
+}
+
+done_testing();
--
2.34.1
[application/x-patch] v39-0002-Add-XID-age-based-replication-slot-invalidation.patch (27.4K, ../../CALj2ACVY+Fd5vC0VjW=5VDK9mmt-Y+PDZxnBp8ngGAZc24Vv9g@mail.gmail.com/3-v39-0002-Add-XID-age-based-replication-slot-invalidation.patch)
download | inline diff:
From c6cee7b246583c05e55b1ed5b14d4d786c2d8ddd Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Fri, 12 Apr 2024 15:00:05 +0000
Subject: [PATCH v39 2/2] Add XID age based replication slot invalidation.
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set an XID age (age of
slot's xmin or catalog_xmin) of say 1 or 1.5 billion, after which
the slots get invalidated.
To achieve the above, postgres introduces a GUC allowing users
set slot XID age. The replication slots whose xmin or catalog_xmin
has reached the age specified by this setting get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/20240327150557.GA3994937%40nathanxps13
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoaRECcnyqxAxUhP5dk2S4HX%3DpGh-p-PkA3uc%2BjG_9hiMw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 26 ++
doc/src/sgml/system-views.sgml | 8 +
src/backend/replication/slot.c | 160 +++++++++-
src/backend/utils/misc/guc_tables.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 3 +
src/test/recovery/t/050_invalidate_slots.pl | 296 +++++++++++++++++-
7 files changed, 490 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a73677b98b..f7aee4663f 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4580,6 +4580,32 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-xid-age" xreflabel="replication_slot_xid_age">
+ <term><varname>replication_slot_xid_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_xid_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots whose <literal>xmin</literal> (the oldest
+ transaction that this slot needs the database to retain) or
+ <literal>catalog_xmin</literal> (the oldest transaction affecting the
+ system catalogs that this slot needs the database to retain) has reached
+ the age specified by this setting. A value of zero (which is default)
+ disables this feature. Users can set this value anywhere from zero to
+ two billion. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command
+ line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during a checkpoint.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 063638beda..05a11a0fe3 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2587,6 +2587,14 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>xid_aged</literal> means that the slot's
+ <literal>xmin</literal> or <literal>catalog_xmin</literal>
+ has reached the age specified by
+ <xref linkend="guc-replication-slot-xid-age"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 7cfbc2dfff..2029efe5a6 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -108,10 +108,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
[RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
+ [RS_INVAL_XID_AGE] = "xid_aged",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
+#define RS_INVAL_MAX_CAUSES RS_INVAL_XID_AGE
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -142,6 +143,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
int max_replication_slots = 10; /* the maximum number of replication
* slots */
int replication_slot_inactive_timeout = 0;
+int replication_slot_xid_age = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -160,6 +162,9 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool ReplicationSlotIsXIDAged(ReplicationSlot *slot,
+ TransactionId *xmin,
+ TransactionId *catalog_xmin);
static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReplicationSlot *s,
@@ -636,8 +641,8 @@ retry:
* it gets invalidated now or has been invalidated previously, because
* there's no use in acquiring the invalidated slot.
*
- * XXX: Currently we check for inactive_timeout invalidation here. We
- * might need to check for other invalidations too.
+ * XXX: Currently we check for inactive_timeout and xid_aged invalidations
+ * here. We might need to check for other invalidations too.
*/
if (check_for_invalidation)
{
@@ -648,6 +653,22 @@ retry:
InvalidTransactionId,
&invalidated);
+ if (!invalidated && released_lock)
+ {
+ /* The slot is still ours */
+ Assert(s->active_pid == MyProcPid);
+
+ /* Reacquire the ControlLock */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ released_lock = false;
+ }
+
+ if (!invalidated)
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_XID_AGE,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ &invalidated);
+
/*
* If the slot has been invalidated, recalculate the resource limits.
*/
@@ -657,7 +678,8 @@ retry:
ReplicationSlotsComputeRequiredLSN();
}
- if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT ||
+ s->data.invalidated == RS_INVAL_XID_AGE)
{
/*
* Release the lock if it's not yet to keep the cleanup path on
@@ -665,7 +687,10 @@ retry:
*/
if (!released_lock)
LWLockRelease(ReplicationSlotControlLock);
+ }
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
Assert(s->inactive_since > 0);
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -675,6 +700,20 @@ retry:
timestamptz_to_str(s->inactive_since),
replication_slot_inactive_timeout)));
}
+
+ if (s->data.invalidated == RS_INVAL_XID_AGE)
+ {
+ Assert(TransactionIdIsValid(s->data.xmin) ||
+ TransactionIdIsValid(s->data.catalog_xmin));
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("The slot's xmin %u or catalog_xmin %u has reached the age %d specified by replication_slot_xid_age.",
+ s->data.xmin,
+ s->data.catalog_xmin,
+ replication_slot_xid_age)));
+ }
}
if (!released_lock)
@@ -1542,7 +1581,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
TransactionId snapshotConflictHorizon,
- TimestampTz inactive_since)
+ TimestampTz inactive_since,
+ TransactionId xmin,
+ TransactionId catalog_xmin)
{
StringInfoData err_detail;
bool hint = false;
@@ -1579,6 +1620,27 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
timestamptz_to_str(inactive_since),
replication_slot_inactive_timeout);
break;
+ case RS_INVAL_XID_AGE:
+ Assert(TransactionIdIsValid(xmin) ||
+ TransactionIdIsValid(catalog_xmin));
+
+ if (TransactionIdIsValid(xmin))
+ {
+ appendStringInfo(&err_detail, _("The slot's xmin %u has reached the age %d specified by replication_slot_xid_age."),
+ xmin,
+ replication_slot_xid_age);
+ break;
+ }
+
+ if (TransactionIdIsValid(catalog_xmin))
+ {
+ appendStringInfo(&err_detail, _("The slot's catalog_xmin %u has reached the age %d specified by replication_slot_xid_age."),
+ catalog_xmin,
+ replication_slot_xid_age);
+ break;
+ }
+
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1623,6 +1685,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
TimestampTz inactive_since = 0;
+ TransactionId aged_xmin = InvalidTransactionId;
+ TransactionId aged_catalog_xmin = InvalidTransactionId;
for (;;)
{
@@ -1739,6 +1803,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
Assert(s->active_pid == 0);
}
break;
+ case RS_INVAL_XID_AGE:
+ if (ReplicationSlotIsXIDAged(s, &aged_xmin, &aged_catalog_xmin))
+ {
+ Assert(TransactionIdIsValid(aged_xmin) ||
+ TransactionIdIsValid(aged_catalog_xmin));
+
+ invalidation_cause = cause;
+ break;
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1827,7 +1901,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
oldestLSN, snapshotConflictHorizon,
- inactive_since);
+ inactive_since, aged_xmin,
+ aged_catalog_xmin);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1874,7 +1949,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
oldestLSN, snapshotConflictHorizon,
- inactive_since);
+ inactive_since, aged_xmin,
+ aged_catalog_xmin);
/* done with this slot for now */
break;
@@ -1898,6 +1974,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
* - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
+ * - RS_INVAL_XID_AGE: slot's xmin or catalog_xmin has reached the age
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -2031,14 +2108,20 @@ CheckPointReplicationSlots(bool is_shutdown)
*
* - Avoid saving slot info to disk two times for each invalidated slot.
*
- * XXX: Should we move inactive_timeout inavalidation check closer to
- * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ * XXX: Should we move inactive_timeout and xid_aged inavalidation checks
+ * closer to wal_removed in CreateCheckPoint and CreateRestartPoint?
*/
invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
0,
InvalidOid,
InvalidTransactionId);
+ if (!invalidated)
+ invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_XID_AGE,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+
if (invalidated)
{
/*
@@ -2050,6 +2133,65 @@ CheckPointReplicationSlots(bool is_shutdown)
}
}
+/*
+ * Returns true if the given replication slot's xmin or catalog_xmin age is
+ * more than replication_slot_xid_age.
+ *
+ * Note that the caller must hold the replication slot's spinlock to avoid
+ * race conditions while this function reads xmin and catalog_xmin.
+ */
+static bool
+ReplicationSlotIsXIDAged(ReplicationSlot *slot, TransactionId *xmin,
+ TransactionId *catalog_xmin)
+{
+ TransactionId cutoff;
+ TransactionId curr;
+
+ if (replication_slot_xid_age == 0)
+ return false;
+
+ curr = ReadNextTransactionId();
+
+ /*
+ * Replication slot's xmin and catalog_xmin can never be larger than the
+ * current transaction id even in the case of transaction ID wraparound.
+ */
+ Assert(slot->data.xmin <= curr);
+ Assert(slot->data.catalog_xmin <= curr);
+
+ /*
+ * The cutoff can tell how far we can go back from the current transaction
+ * id till the age. And then, we check whether or not the xmin or
+ * catalog_xmin falls within the cutoff; if yes, return true, otherwise
+ * false.
+ */
+ cutoff = curr - replication_slot_xid_age;
+
+ if (!TransactionIdIsNormal(cutoff))
+ {
+ cutoff = FirstNormalTransactionId;
+ }
+
+ *xmin = InvalidTransactionId;
+ *catalog_xmin = InvalidTransactionId;
+
+ if (TransactionIdIsNormal(slot->data.xmin) &&
+ TransactionIdPrecedesOrEquals(slot->data.xmin, cutoff))
+ {
+ *xmin = slot->data.xmin;
+ return true;
+ }
+
+ if (TransactionIdIsNormal(slot->data.catalog_xmin) &&
+ TransactionIdPrecedesOrEquals(slot->data.catalog_xmin, cutoff))
+ {
+ *catalog_xmin = slot->data.catalog_xmin;
+ return true;
+ }
+
+ return false;
+}
+
/*
* Load all replication slots from disk into memory at server startup. This
* needs to be run before we start crash recovery.
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 79e7637ec9..ea70e83350 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2994,6 +2994,16 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_xid_age", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Age of the transaction ID at which a replication slot gets invalidated."),
+ gettext_noop("The transaction is the oldest transaction (including the one affecting the system catalogs) that a replication slot needs the database to retain.")
+ },
+ &replication_slot_xid_age,
+ 0, 0, 2000000000,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 819310b0a7..a2387ebd33 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
+#replication_slot_xid_age = 0
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 8727b7b58b..19e5dbfb36 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -55,6 +55,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_WAL_LEVEL,
/* inactive slot timeout has occurred */
RS_INVAL_INACTIVE_TIMEOUT,
+ /* slot's xmin or catalog_xmin has reached the age */
+ RS_INVAL_XID_AGE,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -233,6 +235,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *standby_slot_names;
extern PGDLLIMPORT int replication_slot_inactive_timeout;
+extern PGDLLIMPORT int replication_slot_xid_age;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
index 4663019c16..da05350df4 100644
--- a/src/test/recovery/t/050_invalidate_slots.pl
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -89,7 +89,7 @@ $primary->reload;
# that nobody has acquired that slot yet, so due to
# replication_slot_inactive_timeout setting above it must get invalidated.
wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Set timeout on the standby also to check the synced slots don't get
# invalidated due to timeout on the standby.
@@ -129,7 +129,7 @@ $standby1->stop;
# Wait for the standby's replication slot to become inactive
wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Testcase end: Invalidate streaming standby's slot as well as logical failover
# slot on primary due to replication_slot_inactive_timeout. Also, check the
@@ -197,15 +197,280 @@ $subscriber->stop;
# Wait for the replication slot to become inactive and then invalidated due to
# timeout.
wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Testcase end: Invalidate logical subscriber's slot due to
# replication_slot_inactive_timeout.
# =============================================================================
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot due to replication_slot_xid_age
+# GUC.
+
+# Prepare for the next test
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$primary->reload;
+
+# Create a standby linking to the primary using the replication slot
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+# Enable hs_feedback. The slot should gain an xmin. We set the status interval
+# so we'll see the results promptly.
+$standby2->append_conf(
+ 'postgresql.conf', q{
+primary_slot_name = 'sb2_slot'
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+});
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb2_slot', immediately_reserve := true);
+]);
+
+$standby2->start;
+
+# Create some content on primary to move xmin
+$primary->safe_psql('postgres',
+ "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby2);
+
+$primary->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NOT NULL AND catalog_xmin IS NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'sb2_slot';
+]) or die "Timed out waiting for slot sb2_slot xmin to advance";
+
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$primary->reload;
+
+# Stop standby to make the replication slot's xmin on primary to age
+$standby2->stop;
+
+$logstart = -s $primary->logfile;
+
+# Do some work to advance xids on primary
+advance_xids($primary, 'tab_int');
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($primary, 'sb2_slot', $logstart, 0, 'xid_aged');
+
+# Testcase end: Invalidate streaming standby's slot due to replication_slot_xid_age
+# GUC.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_xid_age GUC.
+
+$publisher = $primary;
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$publisher->reload;
+
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+));
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl2 (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl2 (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl2 VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+$publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE test_tbl2");
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (slot_name = 'lsub2_slot')"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub2');
+
+$result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl2");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NULL AND catalog_xmin IS NOT NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'lsub2_slot';
+]) or die "Timed out waiting for slot lsub2_slot catalog_xmin to advance";
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Do some work to advance xids on publisher
+advance_xids($publisher, 'test_tbl2');
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($publisher, 'lsub2_slot', $logstart, 0,
+ 'xid_aged');
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_xid_age GUC.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical slot on standby that's being synced from
+# the primary due to replication_slot_xid_age GUC.
+
+$publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 0;
+]);
+$publisher->reload;
+
+# Create a standby linking to the primary using the replication slot
+my $standby3 = PostgreSQL::Test::Cluster->new('standby3');
+$standby3->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+$standby3->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb3_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb3_slot', immediately_reserve := true);
+]);
+
+$standby3->start;
+
+my $standby3_logstart = -s $standby3->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby3);
+
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+));
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl3 (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl3 (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl3 VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+$publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE test_tbl3");
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (slot_name = 'lsub3_sync_slot', failover = true)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub3');
+
+$result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl3");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NULL AND catalog_xmin IS NOT NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'lsub3_sync_slot';
+])
+ or die "Timed out waiting for slot lsub3_sync_slot catalog_xmin to advance";
+
+# Synchronize the primary server slots to the standby
+$standby3->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced' and has got catalog_xmin from the primary.
+is( $standby3->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub3_sync_slot' AND synced AND NOT temporary AND
+ xmin IS NULL AND catalog_xmin IS NOT NULL;}
+ ),
+ "t",
+ 'logical slot has synced as true on standby');
+
+my $primary_catalog_xmin = $primary->safe_psql('postgres',
+ "SELECT catalog_xmin FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND catalog_xmin IS NOT NULL;"
+);
+
+my $stabdby3_catalog_xmin = $standby3->safe_psql('postgres',
+ "SELECT catalog_xmin FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND catalog_xmin IS NOT NULL;"
+);
+
+is($primary_catalog_xmin, $stabdby3_catalog_xmin,
+ "check catalog_xmin are same for primary slot and synced slot");
+
+# Enable XID age based invalidation on the standby. Note that we disabled the
+# same on the primary to check if the invalidation occurs for synced slot on
+# the standby.
+$standby3->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$standby3->reload;
+
+$logstart = -s $standby3->logfile;
+
+# Do some work to advance xids on primary
+advance_xids($primary, 'test_tbl3');
+
+# Wait for standby to catch up with the above work
+$primary->wait_for_catchup($standby3);
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($standby3, 'lsub3_sync_slot', $logstart, 0,
+ 'xid_aged');
+
+# Note that the replication slot on the primary is still active
+$result = $primary->safe_psql('postgres',
+ "SELECT COUNT(slot_name) = 1 FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND invalidation_reason IS NULL;"
+);
+
+is($result, 't', "check lsub3_sync_slot is still active on primary");
+
+# Testcase end: Invalidate logical slot on standby that's being synced from
+# the primary due to replication_slot_xid_age GUC.
+# =============================================================================
+
sub wait_for_slot_invalidation
{
- my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my ($node, $slot_name, $offset, $inactive_timeout, $reason) = @_;
my $name = $node->name;
# Wait for the replication slot to become inactive
@@ -238,7 +503,7 @@ sub wait_for_slot_invalidation
'postgres', qq[
SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
WHERE slot_name = '$slot_name' AND
- invalidation_reason = 'inactive_timeout';
+ invalidation_reason = '$reason';
])
or die
"Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
@@ -283,4 +548,25 @@ sub check_for_slot_invalidation_in_server_log
);
}
+# Do some work for advancing xids on a given node
+sub advance_xids
+{
+ my ($node, $table_name) = @_;
+
+ $node->safe_psql(
+ 'postgres', qq[
+ do \$\$
+ begin
+ for i in 10000..11000 loop
+ -- use an exception block so that each iteration eats an XID
+ begin
+ insert into $table_name values (i);
+ exception
+ when division_by_zero then null;
+ end;
+ end loop;
+ end\$\$;
+ ]);
+}
+
done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-06-17 12:25 ` Bharath Rupireddy <[email protected]>
2024-06-17 15:09 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nathan Bossart <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
0 siblings, 2 replies; 98+ messages in thread
From: Bharath Rupireddy @ 2024-06-17 12:25 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On Sat, Apr 13, 2024 at 9:36 AM Bharath Rupireddy
<[email protected]> wrote:
>
> There was a point raised by Amit
> https://www.postgresql.org/message-id/CAA4eK1K8wqLsMw6j0hE_SFoWAeo3Kw8UNnMfhsWaYDF1GWYQ%2Bg%40mail.g...
> on when to do the XID age based invalidation - whether in checkpointer
> or when vacuum is being run or whenever ComputeXIDHorizons gets called
> or in autovacuum process. For now, I've chosen the design to do these
> new invalidation checks in two places - 1) whenever the slot is
> acquired and the slot acquisition errors out if invalidated, 2) during
> checkpoint. However, I'm open to suggestions on this.
Here are my thoughts on when to do the XID age invalidation. In all
the patches sent so far, the XID age invalidation happens in two
places - one during the slot acquisition, and another during the
checkpoint. As the suggestion is to do it during the vacuum (manual
and auto), so that even if the checkpoint isn't happening in the
database for whatever reasons, a vacuum command or autovacuum can
invalidate the slots whose XID is aged.
An idea is to check for XID age based invalidation for all the slots
in ComputeXidHorizons() before it reads replication_slot_xmin and
replication_slot_catalog_xmin, and obviously before the proc array
lock is acquired. A potential problem with this approach is that the
invalidation check can become too aggressive as XID horizons are
computed from many places.
Another idea is to check for XID age based invalidation for all the
slots in higher levels than ComputeXidHorizons(), for example in
vacuum() which is an entry point for both vacuum command and
autovacuum. This approach seems similar to vacuum_failsafe_age GUC
which checks each relation for the failsafe age before vacuum gets
triggered on it.
Does anyone see any issues or risks with the above two approaches or
have any other ideas? Thoughts?
I attached v40 patches here. I reworded some of the ERROR messages,
and did some code clean-up. Note that I haven't implemented any of the
above approaches yet.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/x-patch] v40-0001-Add-inactive_timeout-based-replication-slot-inva.patch (33.6K, ../../CALj2ACX0FVrx9f0c2g19Rb-WAJhm3wdyttde3YQtkotL6bUveA@mail.gmail.com/2-v40-0001-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From 7a920d4f1a4d6a10776ff597d6d931b10340417c Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Mon, 17 Jun 2024 06:55:27 +0000
Subject: [PATCH v40 1/2] Add inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set a timeout of say 1
or 2 or 3 days, after which the inactive slots get invalidated.
To achieve the above, postgres introduces a GUC allowing users
set inactive timeout. The replication slots that are inactive
for longer than specified amount of time get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Note that this new invalidation mechanism won't kick-in for the
slots that are currently being synced from the primary to the
standby, because such synced slots are typically considered not
active (for them to be later considered as inactive) as they don't
perform logical decoding to produce the changes.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/202403260841.5jcv7ihniccy%40alvherre.pgsql
---
doc/src/sgml/config.sgml | 33 ++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 11 +-
src/backend/replication/slot.c | 188 +++++++++++-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 6 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 286 ++++++++++++++++++
13 files changed, 535 insertions(+), 20 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 698169afdb..a01f6d2d14 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4554,6 +4554,39 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during a checkpoint. The time since the slot has become
+ inactive is known from its
+ <structfield>inactive_since</structfield> value using which the
+ timeout is measured.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 8c18bea902..4867af1b61 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2580,6 +2580,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for the duration specified by
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 56d3fb5d0e..a5967d400f 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -448,7 +448,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -651,6 +651,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
" name slot \"%s\" already exists on the standby",
remote_slot->name));
+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to avoid slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
+
/*
* The slot has been synchronized before.
*
@@ -667,7 +674,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 564cfee127..e80872f27b 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -159,6 +161,13 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
+ ReplicationSlot *s,
+ XLogRecPtr oldestLSN,
+ Oid dboid,
+ TransactionId snapshotConflictHorizon,
+ bool *invalidated);
+
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
static void CreateSlotOnDisk(ReplicationSlot *slot);
@@ -535,12 +544,17 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if check_for_invalidation is true and the slot gets
+ * invalidated now or has been invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation)
{
ReplicationSlot *s;
int active_pid;
+ bool released_lock = false;
Assert(name != NULL);
@@ -615,6 +629,57 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ /*
+ * Check if the acquired slot needs to be invalidated. And, error out if
+ * it gets invalidated now or has been invalidated previously, because
+ * there's no use in acquiring the invalidated slot.
+ *
+ * XXX: Currently we check for inactive_timeout invalidation here. We
+ * might need to check for other invalidations too.
+ */
+ if (check_for_invalidation)
+ {
+ bool invalidated = false;
+
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_INACTIVE_TIMEOUT,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ &invalidated);
+
+ /*
+ * If the slot has been invalidated, recalculate the resource limits.
+ */
+ if (invalidated)
+ {
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
+
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ /*
+ * Release the lock if it's not yet to keep the cleanup path on
+ * error happy.
+ */
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive since %s for more than %d seconds specified by \"replication_slot_inactive_timeout\".",
+ timestamptz_to_str(s->inactive_since),
+ replication_slot_inactive_timeout)));
+ }
+ }
+
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +850,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -808,7 +873,7 @@ ReplicationSlotAlter(const char *name, bool failover)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1480,7 +1545,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1510,6 +1576,13 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for more than %d seconds specified by \"replication_slot_inactive_timeout\"."),
+ timestamptz_to_str(inactive_since),
+ replication_slot_inactive_timeout);
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1553,6 +1626,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1560,6 +1634,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1570,6 +1645,18 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced)))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1623,6 +1710,39 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Quick exit if inactive timeout invalidation mechanism
+ * is disabled or slot is currently being used or the slot
+ * on standby is currently being synced from the primary.
+ *
+ * Note that we don't invalidate synced slots because,
+ * they are typically considered not active as they don't
+ * perform logical decoding to produce the changes.
+ */
+ if (replication_slot_inactive_timeout == 0 ||
+ s->inactive_since == 0 ||
+ (RecoveryInProgress() && s->data.synced))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1648,11 +1768,14 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so or if the slot is already ours,
+ * then mark it invalidated. Otherwise we'll signal the owning
+ * process, below, and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot != NULL &&
+ MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1707,7 +1830,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1753,7 +1877,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1776,6 +1901,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1828,7 +1954,7 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk and invalidate slots.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1839,6 +1965,7 @@ void
CheckPointReplicationSlots(bool is_shutdown)
{
int i;
+ bool invalidated = false;
elog(DEBUG1, "performing replication slot checkpoint");
@@ -1886,6 +2013,43 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ elog(DEBUG1, "performing replication slot invalidation");
+
+ /*
+ * Note that we will make another pass over replication slots for
+ * invalidations to keep the code simple. The assumption here is that the
+ * traversal over replication slots isn't that costly even with hundreds
+ * of replication slots. If it ever turns out that this assumption is
+ * wrong, we might have to put the invalidation check logic in the above
+ * loop, for that we might have to do the following:
+ *
+ * - Acqure ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated slot.
+ *
+ * XXX: Should we move inactive_timeout inavalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+
+ if (invalidated)
+ {
+ /*
+ * If any slots have been invalidated, recalculate the resource
+ * limits.
+ */
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index dd6c1d5a7e..9ad3e55704 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -539,7 +539,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c623b07cf0..1741e09259 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 46c258be28..4990e73c97 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2961,6 +2961,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e0567de219..535fb07385 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 1bc80960ef..56d20e1a78 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *standby_slot_names;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -245,7 +248,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait);
extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, bool failover);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..b4c5ce2875 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -10,6 +10,7 @@ tests += {
'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
},
'tests': [
+ 't/050_invalidate_slots.pl',
't/001_stream_rep.pl',
't/002_archiving.pl',
't/003_recovery_targets.pl',
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..4663019c16
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,286 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to replication_slot_inactive_timeout. Also,
+# check the logical failover slot synced on to the standby doesn't invalidate
+# the slot on its own, but gets the invalidated state from the remote slot on
+# the primary.
+
+# Initialize primary node
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid checkpoint during the test, otherwise, the test can get unpredictable
+$primary->append_conf(
+ 'postgresql.conf', q{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create a standby linking to the primary using the replication slot
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr_1 = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb1_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+my $standby1_logstart = -s $standby1->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Synchronize the primary server slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced'.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot lsub1_sync_slot has synced as true on standby');
+
+my $logstart = -s $primary->logfile;
+my $inactive_timeout = 2;
+
+# Set timeout so that the next checkpoint will invalidate the inactive
+# replication slot.
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for the logical failover slot to become inactive on the primary. Note
+# that nobody has acquired that slot yet, so due to
+# replication_slot_inactive_timeout setting above it must get invalidated.
+wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
+ $inactive_timeout);
+
+# Set timeout on the standby also to check the synced slots don't get
+# invalidated due to timeout on the standby.
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$standby1->reload;
+
+# Now, sync the logical failover slot from the remote slot on the primary.
+# Note that the remote slot has already been invalidated due to inactive
+# timeout. Now, the standby must also see it as invalidated.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby";
+
+# Synced slot mustn't get invalidated on the standby even after a checkpoint,
+# it must sync invalidation from the primary. So, we must not see the slot's
+# invalidation message in server log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned slot lsub1_sync_slot has not been invalidated on the standby'
+);
+
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate streaming standby's slot as well as logical failover
+# slot on primary due to replication_slot_inactive_timeout. Also, check the
+# logical failover slot synced on to the standby doesn't invalidate the slot on
+# its own, but gets the invalidated state from the remote slot on the primary.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+
+my $publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber node
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
+
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for slot $slot_name to become inactive on node $name";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of slot $slot_name to be updated on node $name";
+
+ # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
+ # for the slot to get invalidated.
+ sleep($inactive_timeout);
+
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
+
+ # Check that the invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot_name', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot_name"/,
+ "detected error upon trying to acquire invalidated slot $slot_name on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot_name on node $name";
+}
+
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $name = $node->name;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot_name\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot_name invalidation has been logged on node $name"
+ );
+}
+
+done_testing();
--
2.34.1
[application/x-patch] v40-0002-Add-XID-age-based-replication-slot-invalidation.patch (27.3K, ../../CALj2ACX0FVrx9f0c2g19Rb-WAJhm3wdyttde3YQtkotL6bUveA@mail.gmail.com/3-v40-0002-Add-XID-age-based-replication-slot-invalidation.patch)
download | inline diff:
From 18185885fb3132187a2552116ee143c4518c1c4a Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Mon, 17 Jun 2024 09:54:43 +0000
Subject: [PATCH v40 2/2] Add XID age based replication slot invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set an XID age (age of
slot's xmin or catalog_xmin) of say 1 or 1.5 billion, after which
the slots get invalidated.
To achieve the above, postgres introduces a GUC allowing users
set slot XID age. The replication slots whose xmin or catalog_xmin
has reached the age specified by this setting get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/20240327150557.GA3994937%40nathanxps13
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoaRECcnyqxAxUhP5dk2S4HX%3DpGh-p-PkA3uc%2BjG_9hiMw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 26 ++
doc/src/sgml/system-views.sgml | 8 +
src/backend/replication/slot.c | 151 ++++++++-
src/backend/utils/misc/guc_tables.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 3 +
src/test/recovery/t/050_invalidate_slots.pl | 296 +++++++++++++++++-
7 files changed, 481 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a01f6d2d14..114b48e41e 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4587,6 +4587,32 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-xid-age" xreflabel="replication_slot_xid_age">
+ <term><varname>replication_slot_xid_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_xid_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots whose <literal>xmin</literal> (the oldest
+ transaction that this slot needs the database to retain) or
+ <literal>catalog_xmin</literal> (the oldest transaction affecting the
+ system catalogs that this slot needs the database to retain) has reached
+ the age specified by this setting. A value of zero (which is default)
+ disables this feature. Users can set this value anywhere from zero to
+ two billion. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command
+ line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during a checkpoint.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 4867af1b61..0490f9f156 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2587,6 +2587,14 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>xid_aged</literal> means that the slot's
+ <literal>xmin</literal> or <literal>catalog_xmin</literal>
+ has reached the age specified by
+ <xref linkend="guc-replication-slot-xid-age"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index e80872f27b..79ac412d8e 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -108,10 +108,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
[RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
+ [RS_INVAL_XID_AGE] = "xid_aged",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
+#define RS_INVAL_MAX_CAUSES RS_INVAL_XID_AGE
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -142,6 +143,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
int max_replication_slots = 10; /* the maximum number of replication
* slots */
int replication_slot_inactive_timeout = 0;
+int replication_slot_xid_age = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -160,6 +162,9 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool ReplicationSlotIsXIDAged(ReplicationSlot *slot,
+ TransactionId *xmin,
+ TransactionId *catalog_xmin);
static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReplicationSlot *s,
@@ -636,8 +641,8 @@ retry:
* it gets invalidated now or has been invalidated previously, because
* there's no use in acquiring the invalidated slot.
*
- * XXX: Currently we check for inactive_timeout invalidation here. We
- * might need to check for other invalidations too.
+ * XXX: Currently we check for inactive_timeout and xid_aged invalidations
+ * here. We might need to check for other invalidations too.
*/
if (check_for_invalidation)
{
@@ -648,6 +653,22 @@ retry:
InvalidTransactionId,
&invalidated);
+ if (!invalidated && released_lock)
+ {
+ /* The slot is still ours */
+ Assert(s->active_pid == MyProcPid);
+
+ /* Reacquire the ControlLock */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ released_lock = false;
+ }
+
+ if (!invalidated)
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_XID_AGE,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ &invalidated);
+
/*
* If the slot has been invalidated, recalculate the resource limits.
*/
@@ -657,7 +678,8 @@ retry:
ReplicationSlotsComputeRequiredLSN();
}
- if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT ||
+ s->data.invalidated == RS_INVAL_XID_AGE)
{
/*
* Release the lock if it's not yet to keep the cleanup path on
@@ -665,7 +687,10 @@ retry:
*/
if (!released_lock)
LWLockRelease(ReplicationSlotControlLock);
+ }
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
Assert(s->inactive_since > 0);
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -675,6 +700,20 @@ retry:
timestamptz_to_str(s->inactive_since),
replication_slot_inactive_timeout)));
}
+
+ if (s->data.invalidated == RS_INVAL_XID_AGE)
+ {
+ Assert(TransactionIdIsValid(s->data.xmin) ||
+ TransactionIdIsValid(s->data.catalog_xmin));
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("The slot's xmin %u or catalog_xmin %u has reached the age %d specified by \"replication_slot_xid_age\".",
+ s->data.xmin,
+ s->data.catalog_xmin,
+ replication_slot_xid_age)));
+ }
}
if (!released_lock)
@@ -1546,7 +1585,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
TransactionId snapshotConflictHorizon,
- TimestampTz inactive_since)
+ TimestampTz inactive_since,
+ TransactionId xmin,
+ TransactionId catalog_xmin)
{
StringInfoData err_detail;
bool hint = false;
@@ -1583,6 +1624,20 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
timestamptz_to_str(inactive_since),
replication_slot_inactive_timeout);
break;
+ case RS_INVAL_XID_AGE:
+ Assert(TransactionIdIsValid(xmin) ||
+ TransactionIdIsValid(catalog_xmin));
+
+ if (TransactionIdIsValid(xmin))
+ appendStringInfo(&err_detail, _("The slot's xmin %u has reached the age %d specified by \"replication_slot_xid_age\"."),
+ xmin,
+ replication_slot_xid_age);
+ else if (TransactionIdIsValid(catalog_xmin))
+ appendStringInfo(&err_detail, _("The slot's catalog_xmin %u has reached the age %d specified by \"replication_slot_xid_age\"."),
+ catalog_xmin,
+ replication_slot_xid_age);
+
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1627,6 +1682,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
TimestampTz inactive_since = 0;
+ TransactionId aged_xmin = InvalidTransactionId;
+ TransactionId aged_catalog_xmin = InvalidTransactionId;
for (;;)
{
@@ -1743,6 +1800,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
Assert(s->active_pid == 0);
}
break;
+ case RS_INVAL_XID_AGE:
+ if (ReplicationSlotIsXIDAged(s, &aged_xmin, &aged_catalog_xmin))
+ {
+ Assert(TransactionIdIsValid(aged_xmin) ||
+ TransactionIdIsValid(aged_catalog_xmin));
+
+ invalidation_cause = cause;
+ break;
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1831,7 +1898,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
oldestLSN, snapshotConflictHorizon,
- inactive_since);
+ inactive_since, aged_xmin,
+ aged_catalog_xmin);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1878,7 +1946,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
oldestLSN, snapshotConflictHorizon,
- inactive_since);
+ inactive_since, aged_xmin,
+ aged_catalog_xmin);
/* done with this slot for now */
break;
@@ -1902,6 +1971,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
* - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
+ * - RS_INVAL_XID_AGE: slot's xmin or catalog_xmin has reached the age
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -2033,14 +2103,20 @@ CheckPointReplicationSlots(bool is_shutdown)
*
* - Avoid saving slot info to disk two times for each invalidated slot.
*
- * XXX: Should we move inactive_timeout inavalidation check closer to
- * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ * XXX: Should we move inactive_timeout and xid_aged inavalidation checks
+ * closer to wal_removed in CreateCheckPoint and CreateRestartPoint?
*/
invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
0,
InvalidOid,
InvalidTransactionId);
+ if (!invalidated)
+ invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_XID_AGE,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+
if (invalidated)
{
/*
@@ -2052,6 +2128,63 @@ CheckPointReplicationSlots(bool is_shutdown)
}
}
+/*
+ * Returns true if the given replication slot's xmin or catalog_xmin age is
+ * more than replication_slot_xid_age.
+ *
+ * Note that the caller must hold the replication slot's spinlock to avoid
+ * race conditions while this function reads xmin and catalog_xmin.
+ */
+static bool
+ReplicationSlotIsXIDAged(ReplicationSlot *slot, TransactionId *xmin,
+ TransactionId *catalog_xmin)
+{
+ TransactionId cutoff;
+ TransactionId curr;
+
+ if (replication_slot_xid_age == 0)
+ return false;
+
+ curr = ReadNextTransactionId();
+
+ /*
+ * Replication slot's xmin and catalog_xmin can never be larger than the
+ * current transaction id even in the case of transaction ID wraparound.
+ */
+ Assert(slot->data.xmin <= curr);
+ Assert(slot->data.catalog_xmin <= curr);
+
+ /*
+ * The cutoff can tell how far we can go back from the current transaction
+ * id till the age. And then, we check whether or not the xmin or
+ * catalog_xmin falls within the cutoff; if yes, return true, otherwise
+ * false.
+ */
+ cutoff = curr - replication_slot_xid_age;
+
+ if (!TransactionIdIsNormal(cutoff))
+ cutoff = FirstNormalTransactionId;
+
+ *xmin = InvalidTransactionId;
+ *catalog_xmin = InvalidTransactionId;
+
+ if (TransactionIdIsNormal(slot->data.xmin) &&
+ TransactionIdPrecedesOrEquals(slot->data.xmin, cutoff))
+ {
+ *xmin = slot->data.xmin;
+ return true;
+ }
+
+ if (TransactionIdIsNormal(slot->data.catalog_xmin) &&
+ TransactionIdPrecedesOrEquals(slot->data.catalog_xmin, cutoff))
+ {
+ *catalog_xmin = slot->data.catalog_xmin;
+ return true;
+ }
+
+ return false;
+}
+
/*
* Load all replication slots from disk into memory at server startup. This
* needs to be run before we start crash recovery.
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 4990e73c97..ca210c6bf9 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2973,6 +2973,16 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_xid_age", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Age of the transaction ID at which a replication slot gets invalidated."),
+ gettext_noop("The transaction is the oldest transaction (including the one affecting the system catalogs) that a replication slot needs the database to retain.")
+ },
+ &replication_slot_xid_age,
+ 0, 0, 2000000000,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 535fb07385..f04771d65c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
+#replication_slot_xid_age = 0
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 56d20e1a78..e757b836c5 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -55,6 +55,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_WAL_LEVEL,
/* inactive slot timeout has occurred */
RS_INVAL_INACTIVE_TIMEOUT,
+ /* slot's xmin or catalog_xmin has reached the age */
+ RS_INVAL_XID_AGE,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -233,6 +235,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *standby_slot_names;
extern PGDLLIMPORT int replication_slot_inactive_timeout;
+extern PGDLLIMPORT int replication_slot_xid_age;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
index 4663019c16..da05350df4 100644
--- a/src/test/recovery/t/050_invalidate_slots.pl
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -89,7 +89,7 @@ $primary->reload;
# that nobody has acquired that slot yet, so due to
# replication_slot_inactive_timeout setting above it must get invalidated.
wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Set timeout on the standby also to check the synced slots don't get
# invalidated due to timeout on the standby.
@@ -129,7 +129,7 @@ $standby1->stop;
# Wait for the standby's replication slot to become inactive
wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Testcase end: Invalidate streaming standby's slot as well as logical failover
# slot on primary due to replication_slot_inactive_timeout. Also, check the
@@ -197,15 +197,280 @@ $subscriber->stop;
# Wait for the replication slot to become inactive and then invalidated due to
# timeout.
wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Testcase end: Invalidate logical subscriber's slot due to
# replication_slot_inactive_timeout.
# =============================================================================
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot due to replication_slot_xid_age
+# GUC.
+
+# Prepare for the next test
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$primary->reload;
+
+# Create a standby linking to the primary using the replication slot
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+# Enable hs_feedback. The slot should gain an xmin. We set the status interval
+# so we'll see the results promptly.
+$standby2->append_conf(
+ 'postgresql.conf', q{
+primary_slot_name = 'sb2_slot'
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+});
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb2_slot', immediately_reserve := true);
+]);
+
+$standby2->start;
+
+# Create some content on primary to move xmin
+$primary->safe_psql('postgres',
+ "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby2);
+
+$primary->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NOT NULL AND catalog_xmin IS NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'sb2_slot';
+]) or die "Timed out waiting for slot sb2_slot xmin to advance";
+
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$primary->reload;
+
+# Stop standby to make the replication slot's xmin on primary to age
+$standby2->stop;
+
+$logstart = -s $primary->logfile;
+
+# Do some work to advance xids on primary
+advance_xids($primary, 'tab_int');
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($primary, 'sb2_slot', $logstart, 0, 'xid_aged');
+
+# Testcase end: Invalidate streaming standby's slot due to replication_slot_xid_age
+# GUC.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_xid_age GUC.
+
+$publisher = $primary;
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$publisher->reload;
+
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+));
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl2 (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl2 (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl2 VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+$publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE test_tbl2");
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (slot_name = 'lsub2_slot')"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub2');
+
+$result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl2");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NULL AND catalog_xmin IS NOT NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'lsub2_slot';
+]) or die "Timed out waiting for slot lsub2_slot catalog_xmin to advance";
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Do some work to advance xids on publisher
+advance_xids($publisher, 'test_tbl2');
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($publisher, 'lsub2_slot', $logstart, 0,
+ 'xid_aged');
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_xid_age GUC.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical slot on standby that's being synced from
+# the primary due to replication_slot_xid_age GUC.
+
+$publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 0;
+]);
+$publisher->reload;
+
+# Create a standby linking to the primary using the replication slot
+my $standby3 = PostgreSQL::Test::Cluster->new('standby3');
+$standby3->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+$standby3->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb3_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb3_slot', immediately_reserve := true);
+]);
+
+$standby3->start;
+
+my $standby3_logstart = -s $standby3->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby3);
+
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+));
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl3 (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl3 (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl3 VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+$publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE test_tbl3");
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (slot_name = 'lsub3_sync_slot', failover = true)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub3');
+
+$result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl3");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NULL AND catalog_xmin IS NOT NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'lsub3_sync_slot';
+])
+ or die "Timed out waiting for slot lsub3_sync_slot catalog_xmin to advance";
+
+# Synchronize the primary server slots to the standby
+$standby3->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced' and has got catalog_xmin from the primary.
+is( $standby3->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub3_sync_slot' AND synced AND NOT temporary AND
+ xmin IS NULL AND catalog_xmin IS NOT NULL;}
+ ),
+ "t",
+ 'logical slot has synced as true on standby');
+
+my $primary_catalog_xmin = $primary->safe_psql('postgres',
+ "SELECT catalog_xmin FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND catalog_xmin IS NOT NULL;"
+);
+
+my $stabdby3_catalog_xmin = $standby3->safe_psql('postgres',
+ "SELECT catalog_xmin FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND catalog_xmin IS NOT NULL;"
+);
+
+is($primary_catalog_xmin, $stabdby3_catalog_xmin,
+ "check catalog_xmin are same for primary slot and synced slot");
+
+# Enable XID age based invalidation on the standby. Note that we disabled the
+# same on the primary to check if the invalidation occurs for synced slot on
+# the standby.
+$standby3->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$standby3->reload;
+
+$logstart = -s $standby3->logfile;
+
+# Do some work to advance xids on primary
+advance_xids($primary, 'test_tbl3');
+
+# Wait for standby to catch up with the above work
+$primary->wait_for_catchup($standby3);
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($standby3, 'lsub3_sync_slot', $logstart, 0,
+ 'xid_aged');
+
+# Note that the replication slot on the primary is still active
+$result = $primary->safe_psql('postgres',
+ "SELECT COUNT(slot_name) = 1 FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND invalidation_reason IS NULL;"
+);
+
+is($result, 't', "check lsub3_sync_slot is still active on primary");
+
+# Testcase end: Invalidate logical slot on standby that's being synced from
+# the primary due to replication_slot_xid_age GUC.
+# =============================================================================
+
sub wait_for_slot_invalidation
{
- my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my ($node, $slot_name, $offset, $inactive_timeout, $reason) = @_;
my $name = $node->name;
# Wait for the replication slot to become inactive
@@ -238,7 +503,7 @@ sub wait_for_slot_invalidation
'postgres', qq[
SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
WHERE slot_name = '$slot_name' AND
- invalidation_reason = 'inactive_timeout';
+ invalidation_reason = '$reason';
])
or die
"Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
@@ -283,4 +548,25 @@ sub check_for_slot_invalidation_in_server_log
);
}
+# Do some work for advancing xids on a given node
+sub advance_xids
+{
+ my ($node, $table_name) = @_;
+
+ $node->safe_psql(
+ 'postgres', qq[
+ do \$\$
+ begin
+ for i in 10000..11000 loop
+ -- use an exception block so that each iteration eats an XID
+ begin
+ insert into $table_name values (i);
+ exception
+ when division_by_zero then null;
+ end;
+ end loop;
+ end\$\$;
+ ]);
+}
+
done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-06-17 15:09 ` Nathan Bossart <[email protected]>
1 sibling, 0 replies; 98+ messages in thread
From: Nathan Bossart @ 2024-06-17 15:09 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 17, 2024 at 05:55:04PM +0530, Bharath Rupireddy wrote:
> Here are my thoughts on when to do the XID age invalidation. In all
> the patches sent so far, the XID age invalidation happens in two
> places - one during the slot acquisition, and another during the
> checkpoint. As the suggestion is to do it during the vacuum (manual
> and auto), so that even if the checkpoint isn't happening in the
> database for whatever reasons, a vacuum command or autovacuum can
> invalidate the slots whose XID is aged.
+1. IMHO this is a principled choice. The similar max_slot_wal_keep_size
parameter is considered where it arguably matters most: when we are trying
to remove/recycle WAL segments. Since this parameter is intended to
prevent the server from running out of space, it makes sense that we'd
apply it at the point where we are trying to free up space. The proposed
max_slot_xid_age parameter is intended to prevent the server from running
out of transaction IDs, so it follows that we'd apply it at the point where
we reclaim them, which happens to be vacuum.
> An idea is to check for XID age based invalidation for all the slots
> in ComputeXidHorizons() before it reads replication_slot_xmin and
> replication_slot_catalog_xmin, and obviously before the proc array
> lock is acquired. A potential problem with this approach is that the
> invalidation check can become too aggressive as XID horizons are
> computed from many places.
>
> Another idea is to check for XID age based invalidation for all the
> slots in higher levels than ComputeXidHorizons(), for example in
> vacuum() which is an entry point for both vacuum command and
> autovacuum. This approach seems similar to vacuum_failsafe_age GUC
> which checks each relation for the failsafe age before vacuum gets
> triggered on it.
I don't presently have any strong opinion on where this logic should go,
but in general, I think we should only invalidate slots if invalidating
them would allow us to advance the vacuum cutoff. If the cutoff is held
back by something else, I don't see a point in invalidating slots because
we'll just be breaking replication in return for no additional reclaimed
transaction IDs.
--
nathan
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-06-24 06:00 ` Bharath Rupireddy <[email protected]>
2024-07-09 22:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nathan Bossart <[email protected]>
2024-08-12 12:17 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
1 sibling, 3 replies; 98+ messages in thread
From: Bharath Rupireddy @ 2024-06-24 06:00 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On Mon, Jun 17, 2024 at 5:55 PM Bharath Rupireddy
<[email protected]> wrote:
>
> Here are my thoughts on when to do the XID age invalidation. In all
> the patches sent so far, the XID age invalidation happens in two
> places - one during the slot acquisition, and another during the
> checkpoint. As the suggestion is to do it during the vacuum (manual
> and auto), so that even if the checkpoint isn't happening in the
> database for whatever reasons, a vacuum command or autovacuum can
> invalidate the slots whose XID is aged.
>
> An idea is to check for XID age based invalidation for all the slots
> in ComputeXidHorizons() before it reads replication_slot_xmin and
> replication_slot_catalog_xmin, and obviously before the proc array
> lock is acquired. A potential problem with this approach is that the
> invalidation check can become too aggressive as XID horizons are
> computed from many places.
>
> Another idea is to check for XID age based invalidation for all the
> slots in higher levels than ComputeXidHorizons(), for example in
> vacuum() which is an entry point for both vacuum command and
> autovacuum. This approach seems similar to vacuum_failsafe_age GUC
> which checks each relation for the failsafe age before vacuum gets
> triggered on it.
I am attaching the patches implementing the idea of invalidating
replication slots during vacuum when current slot xmin limits
(procArray->replication_slot_xmin and
procArray->replication_slot_catalog_xmin) are aged as per the new XID
age GUC. When either of these limits are aged, there must be at least
one replication slot that is aged, because the xmin limits, after all,
are the minimum of xmin or catalog_xmin of all replication slots. In
this approach, the new XID age GUC will help vacuum when needed,
because the current slot xmin limits are recalculated after
invalidating replication slots that are holding xmins for longer than
the age. The code is placed in vacuum() which is common for both
vacuum command and autovacuum, and gets executed only once every
vacuum cycle to not be too aggressive in invalidating.
However, there might be some concerns with this approach like the following:
1) Adding more code to vacuum might not be acceptable
2) What if invalidation of replication slots emits an error, will it
block vacuum forever? Currently, InvalidateObsoleteReplicationSlots()
is also called as part of the checkpoint, and emitting ERRORs from
within is avoided already. Therefore, there is no concern here for
now.
3) What if there are more replication slots to be invalidated, will it
delay the vacuum? If yes, by how much? <<TODO>>
4) Will the invalidation based on just current replication slot xmin
limits suffice irrespective of vacuum cutoffs? IOW, if the replication
slots are invalidated but vacuum isn't going to do any work because
vacuum cutoffs are not yet met? Is the invalidation work wasteful
here?
5) Is it okay to take just one more time the proc array lock to get
current replication slot xmin limits via
ProcArrayGetReplicationSlotXmin() once every vacuum cycle? <<TODO>>
6) Vacuum command can't be run on the standby in recovery. So, to help
invalidate replication slots on the standby, I have for now let the
checkpointer also do the XID age based invalidation. I know
invalidating both in checkpointer and vacuum may not be a great idea,
but I'm open to thoughts.
Following are some of the alternative approaches which IMHO don't help
vacuum when needed:
a) Let the checkpointer do the XID age based invalidation, and call it
out in the documentation that if the checkpoint doesn't happen, the
new GUC doesn't help even if the vacuum is run. This has been the
approach until v40 patch.
b) Checkpointer and/or other backends add an autovacuum work item via
AutoVacuumRequestWork(), and autovacuum when it gets to it will
invalidate the replication slots. But, what to do for the vacuum
command here?
Please find the attached v41 patches implementing the idea of vacuum
doing the invalidation.
Thoughts?
Thanks to Sawada-san for a detailed off-list discussion.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/x-patch] v41-0001-Add-inactive_timeout-based-replication-slot-inva.patch (33.6K, ../../CALj2ACXQNGXokgx8APwdxrG4MHMF=cOz6XQtUL7EHua9oUfkgA@mail.gmail.com/2-v41-0001-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From dc1eeed06377c11b139724702370ba47cd5d5be3 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Sun, 23 Jun 2024 14:07:25 +0000
Subject: [PATCH v41 1/2] Add inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set a timeout of say 1
or 2 or 3 days, after which the inactive slots get invalidated.
To achieve the above, postgres introduces a GUC allowing users
set inactive timeout. The replication slots that are inactive
for longer than specified amount of time get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Note that this new invalidation mechanism won't kick-in for the
slots that are currently being synced from the primary to the
standby, because such synced slots are typically considered not
active (for them to be later considered as inactive) as they don't
perform logical decoding to produce the changes.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/202403260841.5jcv7ihniccy%40alvherre.pgsql
---
doc/src/sgml/config.sgml | 33 ++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 11 +-
src/backend/replication/slot.c | 188 +++++++++++-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 6 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 286 ++++++++++++++++++
13 files changed, 535 insertions(+), 20 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 0c7a9082c5..5e7a81a1fd 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4554,6 +4554,39 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during a checkpoint. The time since the slot has become
+ inactive is known from its
+ <structfield>inactive_since</structfield> value using which the
+ timeout is measured.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 8c18bea902..4867af1b61 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2580,6 +2580,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for the duration specified by
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 56d3fb5d0e..a5967d400f 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -448,7 +448,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -651,6 +651,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
" name slot \"%s\" already exists on the standby",
remote_slot->name));
+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to avoid slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
+
/*
* The slot has been synchronized before.
*
@@ -667,7 +674,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 564cfee127..e80872f27b 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -159,6 +161,13 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
+ ReplicationSlot *s,
+ XLogRecPtr oldestLSN,
+ Oid dboid,
+ TransactionId snapshotConflictHorizon,
+ bool *invalidated);
+
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
static void CreateSlotOnDisk(ReplicationSlot *slot);
@@ -535,12 +544,17 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if check_for_invalidation is true and the slot gets
+ * invalidated now or has been invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation)
{
ReplicationSlot *s;
int active_pid;
+ bool released_lock = false;
Assert(name != NULL);
@@ -615,6 +629,57 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ /*
+ * Check if the acquired slot needs to be invalidated. And, error out if
+ * it gets invalidated now or has been invalidated previously, because
+ * there's no use in acquiring the invalidated slot.
+ *
+ * XXX: Currently we check for inactive_timeout invalidation here. We
+ * might need to check for other invalidations too.
+ */
+ if (check_for_invalidation)
+ {
+ bool invalidated = false;
+
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_INACTIVE_TIMEOUT,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ &invalidated);
+
+ /*
+ * If the slot has been invalidated, recalculate the resource limits.
+ */
+ if (invalidated)
+ {
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
+
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ /*
+ * Release the lock if it's not yet to keep the cleanup path on
+ * error happy.
+ */
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive since %s for more than %d seconds specified by \"replication_slot_inactive_timeout\".",
+ timestamptz_to_str(s->inactive_since),
+ replication_slot_inactive_timeout)));
+ }
+ }
+
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +850,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -808,7 +873,7 @@ ReplicationSlotAlter(const char *name, bool failover)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1480,7 +1545,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1510,6 +1576,13 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for more than %d seconds specified by \"replication_slot_inactive_timeout\"."),
+ timestamptz_to_str(inactive_since),
+ replication_slot_inactive_timeout);
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1553,6 +1626,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1560,6 +1634,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1570,6 +1645,18 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced)))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1623,6 +1710,39 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Quick exit if inactive timeout invalidation mechanism
+ * is disabled or slot is currently being used or the slot
+ * on standby is currently being synced from the primary.
+ *
+ * Note that we don't invalidate synced slots because,
+ * they are typically considered not active as they don't
+ * perform logical decoding to produce the changes.
+ */
+ if (replication_slot_inactive_timeout == 0 ||
+ s->inactive_since == 0 ||
+ (RecoveryInProgress() && s->data.synced))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1648,11 +1768,14 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so or if the slot is already ours,
+ * then mark it invalidated. Otherwise we'll signal the owning
+ * process, below, and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot != NULL &&
+ MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1707,7 +1830,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1753,7 +1877,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1776,6 +1901,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1828,7 +1954,7 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk and invalidate slots.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1839,6 +1965,7 @@ void
CheckPointReplicationSlots(bool is_shutdown)
{
int i;
+ bool invalidated = false;
elog(DEBUG1, "performing replication slot checkpoint");
@@ -1886,6 +2013,43 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ elog(DEBUG1, "performing replication slot invalidation");
+
+ /*
+ * Note that we will make another pass over replication slots for
+ * invalidations to keep the code simple. The assumption here is that the
+ * traversal over replication slots isn't that costly even with hundreds
+ * of replication slots. If it ever turns out that this assumption is
+ * wrong, we might have to put the invalidation check logic in the above
+ * loop, for that we might have to do the following:
+ *
+ * - Acqure ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated slot.
+ *
+ * XXX: Should we move inactive_timeout inavalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+
+ if (invalidated)
+ {
+ /*
+ * If any slots have been invalidated, recalculate the resource
+ * limits.
+ */
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index dd6c1d5a7e..9ad3e55704 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -539,7 +539,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c623b07cf0..1741e09259 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -846,7 +846,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1459,7 +1459,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 46c258be28..4990e73c97 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2961,6 +2961,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e0567de219..535fb07385 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 1bc80960ef..56d20e1a78 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *standby_slot_names;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -245,7 +248,8 @@ extern void ReplicationSlotDrop(const char *name, bool nowait);
extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, bool failover);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..b4c5ce2875 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -10,6 +10,7 @@ tests += {
'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
},
'tests': [
+ 't/050_invalidate_slots.pl',
't/001_stream_rep.pl',
't/002_archiving.pl',
't/003_recovery_targets.pl',
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..4663019c16
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,286 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to replication_slot_inactive_timeout. Also,
+# check the logical failover slot synced on to the standby doesn't invalidate
+# the slot on its own, but gets the invalidated state from the remote slot on
+# the primary.
+
+# Initialize primary node
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid checkpoint during the test, otherwise, the test can get unpredictable
+$primary->append_conf(
+ 'postgresql.conf', q{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create a standby linking to the primary using the replication slot
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr_1 = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb1_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+my $standby1_logstart = -s $standby1->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Synchronize the primary server slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced'.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot lsub1_sync_slot has synced as true on standby');
+
+my $logstart = -s $primary->logfile;
+my $inactive_timeout = 2;
+
+# Set timeout so that the next checkpoint will invalidate the inactive
+# replication slot.
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for the logical failover slot to become inactive on the primary. Note
+# that nobody has acquired that slot yet, so due to
+# replication_slot_inactive_timeout setting above it must get invalidated.
+wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
+ $inactive_timeout);
+
+# Set timeout on the standby also to check the synced slots don't get
+# invalidated due to timeout on the standby.
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$standby1->reload;
+
+# Now, sync the logical failover slot from the remote slot on the primary.
+# Note that the remote slot has already been invalidated due to inactive
+# timeout. Now, the standby must also see it as invalidated.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby";
+
+# Synced slot mustn't get invalidated on the standby even after a checkpoint,
+# it must sync invalidation from the primary. So, we must not see the slot's
+# invalidation message in server log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned slot lsub1_sync_slot has not been invalidated on the standby'
+);
+
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate streaming standby's slot as well as logical failover
+# slot on primary due to replication_slot_inactive_timeout. Also, check the
+# logical failover slot synced on to the standby doesn't invalidate the slot on
+# its own, but gets the invalidated state from the remote slot on the primary.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+
+my $publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber node
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
+
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for slot $slot_name to become inactive on node $name";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of slot $slot_name to be updated on node $name";
+
+ # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
+ # for the slot to get invalidated.
+ sleep($inactive_timeout);
+
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
+
+ # Check that the invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot_name', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot_name"/,
+ "detected error upon trying to acquire invalidated slot $slot_name on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot_name on node $name";
+}
+
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $name = $node->name;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot_name\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot_name invalidation has been logged on node $name"
+ );
+}
+
+done_testing();
--
2.34.1
[application/x-patch] v41-0002-Add-XID-age-based-replication-slot-invalidation.patch (32.5K, ../../CALj2ACXQNGXokgx8APwdxrG4MHMF=cOz6XQtUL7EHua9oUfkgA@mail.gmail.com/3-v41-0002-Add-XID-age-based-replication-slot-invalidation.patch)
download | inline diff:
From 3e4e9c8965d6109f71318a337c1f1e10f2ab67b6 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Sun, 23 Jun 2024 16:18:21 +0000
Subject: [PATCH v41 2/2] Add XID age based replication slot invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set an XID age (age of
slot's xmin or catalog_xmin) of say 1 or 1.5 billion, after which
the slots get invalidated.
To achieve the above, postgres introduces a GUC allowing users
set slot XID age. The replication slots whose xmin or catalog_xmin
has reached the age specified by this setting get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
- During vacuum (both command-based and autovacuum)
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/20240327150557.GA3994937%40nathanxps13
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoaRECcnyqxAxUhP5dk2S4HX%3DpGh-p-PkA3uc%2BjG_9hiMw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 26 ++
doc/src/sgml/system-views.sgml | 8 +
src/backend/commands/vacuum.c | 80 +++++
src/backend/replication/slot.c | 151 +++++++-
src/backend/utils/misc/guc_tables.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 3 +
src/test/recovery/t/050_invalidate_slots.pl | 321 +++++++++++++++++-
8 files changed, 583 insertions(+), 17 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 5e7a81a1fd..20d800ce0c 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4587,6 +4587,32 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-xid-age" xreflabel="replication_slot_xid_age">
+ <term><varname>replication_slot_xid_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_xid_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots whose <literal>xmin</literal> (the oldest
+ transaction that this slot needs the database to retain) or
+ <literal>catalog_xmin</literal> (the oldest transaction affecting the
+ system catalogs that this slot needs the database to retain) has reached
+ the age specified by this setting. A value of zero (which is default)
+ disables this feature. Users can set this value anywhere from zero to
+ two billion. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command
+ line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during vacuum or during checkpoint.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 4867af1b61..0490f9f156 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2587,6 +2587,14 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>xid_aged</literal> means that the slot's
+ <literal>xmin</literal> or <literal>catalog_xmin</literal>
+ has reached the age specified by
+ <xref linkend="guc-replication-slot-xid-age"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 48f8eab202..9eeb42ac27 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -47,6 +47,7 @@
#include "postmaster/autovacuum.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/interrupt.h"
+#include "replication/slot.h"
#include "storage/bufmgr.h"
#include "storage/lmgr.h"
#include "storage/pmsignal.h"
@@ -116,6 +117,7 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
+static void try_replication_slot_invalidation(void);
/*
* GUC check function to ensure GUC value specified is within the allowable
@@ -452,6 +454,75 @@ ExecVacuum(ParseState *pstate, VacuumStmt *vacstmt, bool isTopLevel)
MemoryContextDelete(vac_context);
}
+/*
+ * Try invalidating replication slots based on current replication slot xmin
+ * limits once every vacuum cycle.
+ */
+static void
+try_replication_slot_invalidation(void)
+{
+ TransactionId min_slot_xmin;
+ TransactionId min_slot_catalog_xmin;
+ bool can_invalidate = false;
+ TransactionId cutoff;
+ TransactionId curr;
+
+ curr = ReadNextTransactionId();
+
+ /*
+ * The cutoff can tell how far we can go back from the current transaction
+ * id till the age. And then, we check whether or not the xmin or
+ * catalog_xmin falls within the cutoff; if yes, return true, otherwise
+ * false.
+ */
+ cutoff = curr - replication_slot_xid_age;
+
+ if (!TransactionIdIsNormal(cutoff))
+ cutoff = FirstNormalTransactionId;
+
+ ProcArrayGetReplicationSlotXmin(&min_slot_xmin, &min_slot_catalog_xmin);
+
+ /*
+ * Current replication slot xmin limits can never be larger than the
+ * current transaction id even in the case of transaction ID wraparound.
+ */
+ Assert(min_slot_xmin <= curr);
+ Assert(min_slot_catalog_xmin <= curr);
+
+ if (TransactionIdIsNormal(min_slot_xmin) &&
+ TransactionIdPrecedesOrEquals(min_slot_xmin, cutoff))
+ can_invalidate = true;
+ else if (TransactionIdIsNormal(min_slot_catalog_xmin) &&
+ TransactionIdPrecedesOrEquals(min_slot_catalog_xmin, cutoff))
+ can_invalidate = true;
+
+ if (can_invalidate)
+ {
+ bool invalidated = false;
+
+ /*
+ * Note that InvalidateObsoleteReplicationSlots is also called as part
+ * of CHECKPOINT, and emitting ERRORs from within is avoided already.
+ * Therefore, there is no concern here that any ERROR from
+ * invalidating replication slots blocks VACUUM.
+ */
+ invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_XID_AGE,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+
+ if (invalidated)
+ {
+ /*
+ * If any slots have been invalidated, recalculate the resource
+ * limits.
+ */
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
+ }
+}
+
/*
* Internal entry point for autovacuum and the VACUUM / ANALYZE commands.
*
@@ -483,6 +554,7 @@ vacuum(List *relations, VacuumParams *params, BufferAccessStrategy bstrategy,
const char *stmttype;
volatile bool in_outer_xact,
use_own_xacts;
+ static bool first_time = true;
Assert(params != NULL);
@@ -594,6 +666,14 @@ vacuum(List *relations, VacuumParams *params, BufferAccessStrategy bstrategy,
CommitTransactionCommand();
}
+ if (params->options & VACOPT_VACUUM &&
+ first_time &&
+ replication_slot_xid_age > 0)
+ {
+ try_replication_slot_invalidation();
+ first_time = false;
+ }
+
/* Turn vacuum cost accounting on or off, and set/clear in_vacuum */
PG_TRY();
{
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index e80872f27b..79ac412d8e 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -108,10 +108,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
[RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
+ [RS_INVAL_XID_AGE] = "xid_aged",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
+#define RS_INVAL_MAX_CAUSES RS_INVAL_XID_AGE
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -142,6 +143,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
int max_replication_slots = 10; /* the maximum number of replication
* slots */
int replication_slot_inactive_timeout = 0;
+int replication_slot_xid_age = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -160,6 +162,9 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool ReplicationSlotIsXIDAged(ReplicationSlot *slot,
+ TransactionId *xmin,
+ TransactionId *catalog_xmin);
static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReplicationSlot *s,
@@ -636,8 +641,8 @@ retry:
* it gets invalidated now or has been invalidated previously, because
* there's no use in acquiring the invalidated slot.
*
- * XXX: Currently we check for inactive_timeout invalidation here. We
- * might need to check for other invalidations too.
+ * XXX: Currently we check for inactive_timeout and xid_aged invalidations
+ * here. We might need to check for other invalidations too.
*/
if (check_for_invalidation)
{
@@ -648,6 +653,22 @@ retry:
InvalidTransactionId,
&invalidated);
+ if (!invalidated && released_lock)
+ {
+ /* The slot is still ours */
+ Assert(s->active_pid == MyProcPid);
+
+ /* Reacquire the ControlLock */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ released_lock = false;
+ }
+
+ if (!invalidated)
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_XID_AGE,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ &invalidated);
+
/*
* If the slot has been invalidated, recalculate the resource limits.
*/
@@ -657,7 +678,8 @@ retry:
ReplicationSlotsComputeRequiredLSN();
}
- if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT ||
+ s->data.invalidated == RS_INVAL_XID_AGE)
{
/*
* Release the lock if it's not yet to keep the cleanup path on
@@ -665,7 +687,10 @@ retry:
*/
if (!released_lock)
LWLockRelease(ReplicationSlotControlLock);
+ }
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
Assert(s->inactive_since > 0);
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -675,6 +700,20 @@ retry:
timestamptz_to_str(s->inactive_since),
replication_slot_inactive_timeout)));
}
+
+ if (s->data.invalidated == RS_INVAL_XID_AGE)
+ {
+ Assert(TransactionIdIsValid(s->data.xmin) ||
+ TransactionIdIsValid(s->data.catalog_xmin));
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("The slot's xmin %u or catalog_xmin %u has reached the age %d specified by \"replication_slot_xid_age\".",
+ s->data.xmin,
+ s->data.catalog_xmin,
+ replication_slot_xid_age)));
+ }
}
if (!released_lock)
@@ -1546,7 +1585,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
TransactionId snapshotConflictHorizon,
- TimestampTz inactive_since)
+ TimestampTz inactive_since,
+ TransactionId xmin,
+ TransactionId catalog_xmin)
{
StringInfoData err_detail;
bool hint = false;
@@ -1583,6 +1624,20 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
timestamptz_to_str(inactive_since),
replication_slot_inactive_timeout);
break;
+ case RS_INVAL_XID_AGE:
+ Assert(TransactionIdIsValid(xmin) ||
+ TransactionIdIsValid(catalog_xmin));
+
+ if (TransactionIdIsValid(xmin))
+ appendStringInfo(&err_detail, _("The slot's xmin %u has reached the age %d specified by \"replication_slot_xid_age\"."),
+ xmin,
+ replication_slot_xid_age);
+ else if (TransactionIdIsValid(catalog_xmin))
+ appendStringInfo(&err_detail, _("The slot's catalog_xmin %u has reached the age %d specified by \"replication_slot_xid_age\"."),
+ catalog_xmin,
+ replication_slot_xid_age);
+
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1627,6 +1682,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
TimestampTz inactive_since = 0;
+ TransactionId aged_xmin = InvalidTransactionId;
+ TransactionId aged_catalog_xmin = InvalidTransactionId;
for (;;)
{
@@ -1743,6 +1800,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
Assert(s->active_pid == 0);
}
break;
+ case RS_INVAL_XID_AGE:
+ if (ReplicationSlotIsXIDAged(s, &aged_xmin, &aged_catalog_xmin))
+ {
+ Assert(TransactionIdIsValid(aged_xmin) ||
+ TransactionIdIsValid(aged_catalog_xmin));
+
+ invalidation_cause = cause;
+ break;
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1831,7 +1898,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
oldestLSN, snapshotConflictHorizon,
- inactive_since);
+ inactive_since, aged_xmin,
+ aged_catalog_xmin);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1878,7 +1946,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
oldestLSN, snapshotConflictHorizon,
- inactive_since);
+ inactive_since, aged_xmin,
+ aged_catalog_xmin);
/* done with this slot for now */
break;
@@ -1902,6 +1971,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
* - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
+ * - RS_INVAL_XID_AGE: slot's xmin or catalog_xmin has reached the age
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -2033,14 +2103,20 @@ CheckPointReplicationSlots(bool is_shutdown)
*
* - Avoid saving slot info to disk two times for each invalidated slot.
*
- * XXX: Should we move inactive_timeout inavalidation check closer to
- * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ * XXX: Should we move inactive_timeout and xid_aged inavalidation checks
+ * closer to wal_removed in CreateCheckPoint and CreateRestartPoint?
*/
invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
0,
InvalidOid,
InvalidTransactionId);
+ if (!invalidated)
+ invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_XID_AGE,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+
if (invalidated)
{
/*
@@ -2052,6 +2128,63 @@ CheckPointReplicationSlots(bool is_shutdown)
}
}
+/*
+ * Returns true if the given replication slot's xmin or catalog_xmin age is
+ * more than replication_slot_xid_age.
+ *
+ * Note that the caller must hold the replication slot's spinlock to avoid
+ * race conditions while this function reads xmin and catalog_xmin.
+ */
+static bool
+ReplicationSlotIsXIDAged(ReplicationSlot *slot, TransactionId *xmin,
+ TransactionId *catalog_xmin)
+{
+ TransactionId cutoff;
+ TransactionId curr;
+
+ if (replication_slot_xid_age == 0)
+ return false;
+
+ curr = ReadNextTransactionId();
+
+ /*
+ * Replication slot's xmin and catalog_xmin can never be larger than the
+ * current transaction id even in the case of transaction ID wraparound.
+ */
+ Assert(slot->data.xmin <= curr);
+ Assert(slot->data.catalog_xmin <= curr);
+
+ /*
+ * The cutoff can tell how far we can go back from the current transaction
+ * id till the age. And then, we check whether or not the xmin or
+ * catalog_xmin falls within the cutoff; if yes, return true, otherwise
+ * false.
+ */
+ cutoff = curr - replication_slot_xid_age;
+
+ if (!TransactionIdIsNormal(cutoff))
+ cutoff = FirstNormalTransactionId;
+
+ *xmin = InvalidTransactionId;
+ *catalog_xmin = InvalidTransactionId;
+
+ if (TransactionIdIsNormal(slot->data.xmin) &&
+ TransactionIdPrecedesOrEquals(slot->data.xmin, cutoff))
+ {
+ *xmin = slot->data.xmin;
+ return true;
+ }
+
+ if (TransactionIdIsNormal(slot->data.catalog_xmin) &&
+ TransactionIdPrecedesOrEquals(slot->data.catalog_xmin, cutoff))
+ {
+ *catalog_xmin = slot->data.catalog_xmin;
+ return true;
+ }
+
+ return false;
+}
+
/*
* Load all replication slots from disk into memory at server startup. This
* needs to be run before we start crash recovery.
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 4990e73c97..ca210c6bf9 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -2973,6 +2973,16 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_xid_age", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Age of the transaction ID at which a replication slot gets invalidated."),
+ gettext_noop("The transaction is the oldest transaction (including the one affecting the system catalogs) that a replication slot needs the database to retain.")
+ },
+ &replication_slot_xid_age,
+ 0, 0, 2000000000,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 535fb07385..f04771d65c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
+#replication_slot_xid_age = 0
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 56d20e1a78..e757b836c5 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -55,6 +55,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_WAL_LEVEL,
/* inactive slot timeout has occurred */
RS_INVAL_INACTIVE_TIMEOUT,
+ /* slot's xmin or catalog_xmin has reached the age */
+ RS_INVAL_XID_AGE,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -233,6 +235,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *standby_slot_names;
extern PGDLLIMPORT int replication_slot_inactive_timeout;
+extern PGDLLIMPORT int replication_slot_xid_age;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
index 4663019c16..18300cfeca 100644
--- a/src/test/recovery/t/050_invalidate_slots.pl
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -89,7 +89,7 @@ $primary->reload;
# that nobody has acquired that slot yet, so due to
# replication_slot_inactive_timeout setting above it must get invalidated.
wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Set timeout on the standby also to check the synced slots don't get
# invalidated due to timeout on the standby.
@@ -129,7 +129,7 @@ $standby1->stop;
# Wait for the standby's replication slot to become inactive
wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Testcase end: Invalidate streaming standby's slot as well as logical failover
# slot on primary due to replication_slot_inactive_timeout. Also, check the
@@ -197,15 +197,280 @@ $subscriber->stop;
# Wait for the replication slot to become inactive and then invalidated due to
# timeout.
wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Testcase end: Invalidate logical subscriber's slot due to
# replication_slot_inactive_timeout.
# =============================================================================
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot due to replication_slot_xid_age
+# GUC.
+
+# Prepare for the next test
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$primary->reload;
+
+# Create a standby linking to the primary using the replication slot
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+# Enable hs_feedback. The slot should gain an xmin. We set the status interval
+# so we'll see the results promptly.
+$standby2->append_conf(
+ 'postgresql.conf', q{
+primary_slot_name = 'sb2_slot'
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+});
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb2_slot', immediately_reserve := true);
+]);
+
+$standby2->start;
+
+# Create some content on primary to move xmin
+$primary->safe_psql('postgres',
+ "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby2);
+
+$primary->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NOT NULL AND catalog_xmin IS NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'sb2_slot';
+]) or die "Timed out waiting for slot sb2_slot xmin to advance";
+
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$primary->reload;
+
+# Stop standby to make the replication slot's xmin on primary to age
+$standby2->stop;
+
+$logstart = -s $primary->logfile;
+
+# Do some work to advance xids on primary
+advance_xids($primary, 'tab_int');
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($primary, 'sb2_slot', $logstart, 0, 'xid_aged');
+
+# Testcase end: Invalidate streaming standby's slot due to replication_slot_xid_age
+# GUC.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_xid_age GUC.
+
+$publisher = $primary;
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$publisher->reload;
+
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+));
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl2 (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl2 (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl2 VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+$publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE test_tbl2");
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (slot_name = 'lsub2_slot')"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub2');
+
+$result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl2");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NULL AND catalog_xmin IS NOT NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'lsub2_slot';
+]) or die "Timed out waiting for slot lsub2_slot catalog_xmin to advance";
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Do some work to advance xids on publisher
+advance_xids($publisher, 'test_tbl2');
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($publisher, 'lsub2_slot', $logstart, 0,
+ 'xid_aged');
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_xid_age GUC.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical slot on standby that's being synced from
+# the primary due to replication_slot_xid_age GUC.
+
+$publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 0;
+]);
+$publisher->reload;
+
+# Create a standby linking to the primary using the replication slot
+my $standby3 = PostgreSQL::Test::Cluster->new('standby3');
+$standby3->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+$standby3->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb3_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb3_slot', immediately_reserve := true);
+]);
+
+$standby3->start;
+
+my $standby3_logstart = -s $standby3->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby3);
+
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+));
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl3 (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl3 (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl3 VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+$publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE test_tbl3");
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (slot_name = 'lsub3_sync_slot', failover = true)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub3');
+
+$result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl3");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NULL AND catalog_xmin IS NOT NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'lsub3_sync_slot';
+])
+ or die "Timed out waiting for slot lsub3_sync_slot catalog_xmin to advance";
+
+# Synchronize the primary server slots to the standby
+$standby3->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced' and has got catalog_xmin from the primary.
+is( $standby3->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub3_sync_slot' AND synced AND NOT temporary AND
+ xmin IS NULL AND catalog_xmin IS NOT NULL;}
+ ),
+ "t",
+ 'logical slot has synced as true on standby');
+
+my $primary_catalog_xmin = $primary->safe_psql('postgres',
+ "SELECT catalog_xmin FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND catalog_xmin IS NOT NULL;"
+);
+
+my $stabdby3_catalog_xmin = $standby3->safe_psql('postgres',
+ "SELECT catalog_xmin FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND catalog_xmin IS NOT NULL;"
+);
+
+is($primary_catalog_xmin, $stabdby3_catalog_xmin,
+ "check catalog_xmin are same for primary slot and synced slot");
+
+# Enable XID age based invalidation on the standby. Note that we disabled the
+# same on the primary to check if the invalidation occurs for synced slot on
+# the standby.
+$standby3->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$standby3->reload;
+
+$logstart = -s $standby3->logfile;
+
+# Do some work to advance xids on primary
+advance_xids($primary, 'test_tbl3');
+
+# Wait for standby to catch up with the above work
+$primary->wait_for_catchup($standby3);
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($standby3, 'lsub3_sync_slot', $logstart, 0,
+ 'xid_aged');
+
+# Note that the replication slot on the primary is still active
+$result = $primary->safe_psql('postgres',
+ "SELECT COUNT(slot_name) = 1 FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND invalidation_reason IS NULL;"
+);
+
+is($result, 't', "check lsub3_sync_slot is still active on primary");
+
+# Testcase end: Invalidate logical slot on standby that's being synced from
+# the primary due to replication_slot_xid_age GUC.
+# =============================================================================
+
sub wait_for_slot_invalidation
{
- my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my ($node, $slot_name, $offset, $inactive_timeout, $reason) = @_;
my $name = $node->name;
# Wait for the replication slot to become inactive
@@ -231,14 +496,15 @@ sub wait_for_slot_invalidation
# for the slot to get invalidated.
sleep($inactive_timeout);
- check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset,
+ $reason);
# Wait for the inactive replication slot to be invalidated
$node->poll_query_until(
'postgres', qq[
SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
WHERE slot_name = '$slot_name' AND
- invalidation_reason = 'inactive_timeout';
+ invalidation_reason = '$reason';
])
or die
"Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
@@ -262,15 +528,33 @@ sub wait_for_slot_invalidation
# Check for invalidation of slot in server log
sub check_for_slot_invalidation_in_server_log
{
- my ($node, $slot_name, $offset) = @_;
+ my ($node, $slot_name, $offset, $reason) = @_;
my $name = $node->name;
my $invalidated = 0;
+ my $isrecovery =
+ $node->safe_psql('postgres', "SELECT pg_is_in_recovery()");
+
+ chomp($isrecovery);
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
- $node->safe_psql('postgres', "CHECKPOINT");
+ if ($reason eq 'xid_aged' && $isrecovery eq 'f')
+ {
+ $node->safe_psql('postgres', "VACUUM");
+ }
+ else
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ }
+
if ($node->log_contains(
"invalidating obsolete replication slot \"$slot_name\"",
+ $offset)
+ || $node->log_contains(
+ "The slot's xmin .* has reached the age .* specified by \"replication_slot_xid_age\".",
+ $offset)
+ || $node->log_contains(
+ "The slot's catalog_xmin .* has reached the age .* specified by \"replication_slot_xid_age\".",
$offset))
{
$invalidated = 1;
@@ -283,4 +567,25 @@ sub check_for_slot_invalidation_in_server_log
);
}
+# Do some work for advancing xids on a given node
+sub advance_xids
+{
+ my ($node, $table_name) = @_;
+
+ $node->safe_psql(
+ 'postgres', qq[
+ do \$\$
+ begin
+ for i in 10000..11000 loop
+ -- use an exception block so that each iteration eats an XID
+ begin
+ insert into $table_name values (i);
+ exception
+ when division_by_zero then null;
+ end;
+ end loop;
+ end\$\$;
+ ]);
+}
+
done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-07-09 22:01 ` Nathan Bossart <[email protected]>
2024-08-12 21:32 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
2 siblings, 1 reply; 98+ messages in thread
From: Nathan Bossart @ 2024-07-09 22:01 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 24, 2024 at 11:30:00AM +0530, Bharath Rupireddy wrote:
> 6) Vacuum command can't be run on the standby in recovery. So, to help
> invalidate replication slots on the standby, I have for now let the
> checkpointer also do the XID age based invalidation. I know
> invalidating both in checkpointer and vacuum may not be a great idea,
> but I'm open to thoughts.
Hm. I hadn't considered this angle.
> a) Let the checkpointer do the XID age based invalidation, and call it
> out in the documentation that if the checkpoint doesn't happen, the
> new GUC doesn't help even if the vacuum is run. This has been the
> approach until v40 patch.
My first reaction is that this is probably okay. I guess you might run
into problems if you set max_slot_xid_age to 2B and checkpoint_timeout to 1
day, but even in that case your transaction ID usage rate would need to be
pretty high for wraparound to occur.
--
nathan
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-07-09 22:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nathan Bossart <[email protected]>
@ 2024-08-12 21:32 ` Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Masahiko Sawada @ 2024-08-12 21:32 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Jul 9, 2024 at 3:01 PM Nathan Bossart <[email protected]> wrote:
>
> On Mon, Jun 24, 2024 at 11:30:00AM +0530, Bharath Rupireddy wrote:
> > 6) Vacuum command can't be run on the standby in recovery. So, to help
> > invalidate replication slots on the standby, I have for now let the
> > checkpointer also do the XID age based invalidation. I know
> > invalidating both in checkpointer and vacuum may not be a great idea,
> > but I'm open to thoughts.
>
> Hm. I hadn't considered this angle.
Another idea would be to let the startup process do slot invalidation
when replaying a RUNNING_XACTS record. Since a RUNNING_XACTS record
has the latest XID on the primary, I think the startup process can
compare it to the slot-xmin, and invalidate slots which are older than
the age limit.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-08-12 12:17 ` Ajin Cherian <[email protected]>
2 siblings, 0 replies; 98+ messages in thread
From: Ajin Cherian @ 2024-08-12 12:17 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 24, 2024 at 4:01 PM Bharath Rupireddy <
[email protected]> wrote:
> Hi,
>
> On Mon, Jun 17, 2024 at 5:55 PM Bharath Rupireddy
> <[email protected]> wrote:
> >
> > Here are my thoughts on when to do the XID age invalidation. In all
> > the patches sent so far, the XID age invalidation happens in two
> > places - one during the slot acquisition, and another during the
> > checkpoint. As the suggestion is to do it during the vacuum (manual
> > and auto), so that even if the checkpoint isn't happening in the
> > database for whatever reasons, a vacuum command or autovacuum can
> > invalidate the slots whose XID is aged.
> >
> > An idea is to check for XID age based invalidation for all the slots
> > in ComputeXidHorizons() before it reads replication_slot_xmin and
> > replication_slot_catalog_xmin, and obviously before the proc array
> > lock is acquired. A potential problem with this approach is that the
> > invalidation check can become too aggressive as XID horizons are
> > computed from many places.
> >
> > Another idea is to check for XID age based invalidation for all the
> > slots in higher levels than ComputeXidHorizons(), for example in
> > vacuum() which is an entry point for both vacuum command and
> > autovacuum. This approach seems similar to vacuum_failsafe_age GUC
> > which checks each relation for the failsafe age before vacuum gets
> > triggered on it.
>
> I am attaching the patches implementing the idea of invalidating
> replication slots during vacuum when current slot xmin limits
> (procArray->replication_slot_xmin and
> procArray->replication_slot_catalog_xmin) are aged as per the new XID
> age GUC. When either of these limits are aged, there must be at least
> one replication slot that is aged, because the xmin limits, after all,
> are the minimum of xmin or catalog_xmin of all replication slots. In
> this approach, the new XID age GUC will help vacuum when needed,
> because the current slot xmin limits are recalculated after
> invalidating replication slots that are holding xmins for longer than
> the age. The code is placed in vacuum() which is common for both
> vacuum command and autovacuum, and gets executed only once every
> vacuum cycle to not be too aggressive in invalidating.
>
> However, there might be some concerns with this approach like the
> following:
> 1) Adding more code to vacuum might not be acceptable
> 2) What if invalidation of replication slots emits an error, will it
> block vacuum forever? Currently, InvalidateObsoleteReplicationSlots()
> is also called as part of the checkpoint, and emitting ERRORs from
> within is avoided already. Therefore, there is no concern here for
> now.
> 3) What if there are more replication slots to be invalidated, will it
> delay the vacuum? If yes, by how much? <<TODO>>
> 4) Will the invalidation based on just current replication slot xmin
> limits suffice irrespective of vacuum cutoffs? IOW, if the replication
> slots are invalidated but vacuum isn't going to do any work because
> vacuum cutoffs are not yet met? Is the invalidation work wasteful
> here?
> 5) Is it okay to take just one more time the proc array lock to get
> current replication slot xmin limits via
> ProcArrayGetReplicationSlotXmin() once every vacuum cycle? <<TODO>>
> 6) Vacuum command can't be run on the standby in recovery. So, to help
> invalidate replication slots on the standby, I have for now let the
> checkpointer also do the XID age based invalidation. I know
> invalidating both in checkpointer and vacuum may not be a great idea,
> but I'm open to thoughts.
>
> Following are some of the alternative approaches which IMHO don't help
> vacuum when needed:
> a) Let the checkpointer do the XID age based invalidation, and call it
> out in the documentation that if the checkpoint doesn't happen, the
> new GUC doesn't help even if the vacuum is run. This has been the
> approach until v40 patch.
> b) Checkpointer and/or other backends add an autovacuum work item via
> AutoVacuumRequestWork(), and autovacuum when it gets to it will
> invalidate the replication slots. But, what to do for the vacuum
> command here?
>
> Please find the attached v41 patches implementing the idea of vacuum
> doing the invalidation.
>
> Thoughts?
>
> Thanks to Sawada-san for a detailed off-list discussion.
>
The patch no longer applies on HEAD, please rebase.
regards,
Ajin Cherian
Fujitsu Australia
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-08-14 03:50 ` Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2 siblings, 1 reply; 98+ messages in thread
From: Ajin Cherian @ 2024-08-14 03:50 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Jun 24, 2024 at 4:01 PM Bharath Rupireddy <
[email protected]> wrote:
> Hi,
>
> On Mon, Jun 17, 2024 at 5:55 PM Bharath Rupireddy
> <[email protected]> wrote:
>
> Please find the attached v41 patches implementing the idea of vacuum
> doing the invalidation.
>
> Thoughts?
>
>
>
Some minor comments on the patch:
1.
+ /*
+ * Release the lock if it's not yet to keep the cleanup path on
+ * error happy.
+ */
I suggest rephrasing to: " "Release the lock if it hasn't been already to
ensure smooth cleanup on error."
2.
elog(DEBUG1, "performing replication slot invalidation");
Probably change it to "performing replication slot invalidation checks" as
we might not actually invalidate any slot here.
3.
In CheckPointReplicationSlots()
+ invalidated =
InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+
+ if (invalidated)
+ {
+ /*
+ * If any slots have been invalidated, recalculate the resource
+ * limits.
+ */
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
Is this calculation of resource limits really required here when the same
is already done inside InvalidateObsoleteReplicationSlots()
regards,
Ajin Cherian
Fujitsu Australia
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
@ 2024-08-26 06:14 ` Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-08-26 06:14 UTC (permalink / raw)
To: Ajin Cherian <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Aug 14, 2024 at 9:20 AM Ajin Cherian <[email protected]> wrote:
>
> Some minor comments on the patch:
Thanks for reviewing.
> 1.
> + /*
> + * Release the lock if it's not yet to keep the cleanup path on
> + * error happy.
> + */
>
> I suggest rephrasing to: " "Release the lock if it hasn't been already to ensure smooth cleanup on error."
Changed.
> 2.
>
> elog(DEBUG1, "performing replication slot invalidation");
>
> Probably change it to "performing replication slot invalidation checks" as we might not actually invalidate any slot here.
Changed.
> 3.
>
> + ReplicationSlotsComputeRequiredXmin(false);
> + ReplicationSlotsComputeRequiredLSN();
> + }
>
> Is this calculation of resource limits really required here when the same is already done inside InvalidateObsoleteReplicationSlots()
Nice catch. Removed.
Please find the attached v42 patches.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/x-patch] v42-0001-Add-inactive_timeout-based-replication-slot-inva.patch (33.2K, ../../CALj2ACVu0EJ+QQ7W0oHbL_+hCtk+p-00MNhE4hfjFcXaMepMfg@mail.gmail.com/2-v42-0001-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From 1a96a7decd928b4c36429a9a6d97467d61f87896 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Mon, 26 Aug 2024 05:22:15 +0000
Subject: [PATCH v42 1/2] Add inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set a timeout of say 1
or 2 or 3 days, after which the inactive slots get invalidated.
To achieve the above, postgres introduces a GUC allowing users
set inactive timeout. The replication slots that are inactive
for longer than specified amount of time get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Note that this new invalidation mechanism won't kick-in for the
slots that are currently being synced from the primary to the
standby, because such synced slots are typically considered not
active (for them to be later considered as inactive) as they don't
perform logical decoding to produce the changes.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila
Reviewed-by: Ajin Cherian, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/202403260841.5jcv7ihniccy%40alvherre.pgsql
---
doc/src/sgml/config.sgml | 33 ++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 11 +-
src/backend/replication/slot.c | 175 ++++++++++-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 6 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 286 ++++++++++++++++++
13 files changed, 522 insertions(+), 20 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 2937384b00..fbbacbee5b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4560,6 +4560,39 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during a checkpoint. The time since the slot has become
+ inactive is known from its
+ <structfield>inactive_since</structfield> value using which the
+ timeout is measured.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 634a4c0fab..9e00f7d184 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2618,6 +2618,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for the duration specified by
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 2d1914ce08..abc2c06feb 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -448,7 +448,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -651,6 +651,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
" name slot \"%s\" already exists on the standby",
remote_slot->name));
+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to avoid slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
+
/*
* The slot has been synchronized before.
*
@@ -667,7 +674,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index c290339af5..5f504f956f 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -159,6 +161,13 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
+ ReplicationSlot *s,
+ XLogRecPtr oldestLSN,
+ Oid dboid,
+ TransactionId snapshotConflictHorizon,
+ bool *invalidated);
+
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
static void CreateSlotOnDisk(ReplicationSlot *slot);
@@ -535,12 +544,17 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if check_for_invalidation is true and the slot gets
+ * invalidated now or has been invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation)
{
ReplicationSlot *s;
int active_pid;
+ bool released_lock = false;
Assert(name != NULL);
@@ -615,6 +629,57 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ /*
+ * Check if the acquired slot needs to be invalidated. And, error out if
+ * it gets invalidated now or has been invalidated previously, because
+ * there's no use in acquiring the invalidated slot.
+ *
+ * XXX: Currently we check for inactive_timeout invalidation here. We
+ * might need to check for other invalidations too.
+ */
+ if (check_for_invalidation)
+ {
+ bool invalidated = false;
+
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_INACTIVE_TIMEOUT,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ &invalidated);
+
+ /*
+ * If the slot has been invalidated, recalculate the resource limits.
+ */
+ if (invalidated)
+ {
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
+
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ /*
+ * Release the lock if it hasn't been already, to ensure smooth
+ * cleanup on error.
+ */
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive since %s for more than %d seconds specified by \"replication_slot_inactive_timeout\".",
+ timestamptz_to_str(s->inactive_since),
+ replication_slot_inactive_timeout)));
+ }
+ }
+
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +850,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +877,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1501,7 +1566,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1531,6 +1597,13 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for more than %d seconds specified by \"replication_slot_inactive_timeout\"."),
+ timestamptz_to_str(inactive_since),
+ replication_slot_inactive_timeout);
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1574,6 +1647,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1581,6 +1655,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1591,6 +1666,18 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced)))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1644,6 +1731,39 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Quick exit if inactive timeout invalidation mechanism
+ * is disabled or slot is currently being used or the slot
+ * on standby is currently being synced from the primary.
+ *
+ * Note that we don't invalidate synced slots because,
+ * they are typically considered not active as they don't
+ * perform logical decoding to produce the changes.
+ */
+ if (replication_slot_inactive_timeout == 0 ||
+ s->inactive_since == 0 ||
+ (RecoveryInProgress() && s->data.synced))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1669,11 +1789,14 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so or if the slot is already ours,
+ * then mark it invalidated. Otherwise we'll signal the owning
+ * process, below, and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot != NULL &&
+ MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1728,7 +1851,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1774,7 +1898,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1797,6 +1922,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1849,7 +1975,7 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk and invalidate slots.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1907,6 +2033,31 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * Note that we will make another pass over replication slots for
+ * invalidations to keep the code simple. The assumption here is that the
+ * traversal over replication slots isn't that costly even with hundreds
+ * of replication slots. If it ever turns out that this assumption is
+ * wrong, we might have to put the invalidation check logic in the above
+ * loop, for that we might have to do the following:
+ *
+ * - Acqure ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated slot.
+ *
+ * XXX: Should we move inactive_timeout inavalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0, InvalidOid, InvalidTransactionId);
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index c7bfbb15e0..b1b7b075bd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -540,7 +540,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c5f1009f37..61a0e38715 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -844,7 +844,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1462,7 +1462,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index af227b1f24..861692c683 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a..deca3a4aeb 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index c2ee149fd6..dd56a77547 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -246,7 +249,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 712924c2fa..301be0f6c1 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -10,6 +10,7 @@ tests += {
'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
},
'tests': [
+ 't/050_invalidate_slots.pl',
't/001_stream_rep.pl',
't/002_archiving.pl',
't/003_recovery_targets.pl',
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..4663019c16
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,286 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to replication_slot_inactive_timeout. Also,
+# check the logical failover slot synced on to the standby doesn't invalidate
+# the slot on its own, but gets the invalidated state from the remote slot on
+# the primary.
+
+# Initialize primary node
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid checkpoint during the test, otherwise, the test can get unpredictable
+$primary->append_conf(
+ 'postgresql.conf', q{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create a standby linking to the primary using the replication slot
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr_1 = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb1_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+my $standby1_logstart = -s $standby1->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Synchronize the primary server slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced'.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot lsub1_sync_slot has synced as true on standby');
+
+my $logstart = -s $primary->logfile;
+my $inactive_timeout = 2;
+
+# Set timeout so that the next checkpoint will invalidate the inactive
+# replication slot.
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for the logical failover slot to become inactive on the primary. Note
+# that nobody has acquired that slot yet, so due to
+# replication_slot_inactive_timeout setting above it must get invalidated.
+wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
+ $inactive_timeout);
+
+# Set timeout on the standby also to check the synced slots don't get
+# invalidated due to timeout on the standby.
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$standby1->reload;
+
+# Now, sync the logical failover slot from the remote slot on the primary.
+# Note that the remote slot has already been invalidated due to inactive
+# timeout. Now, the standby must also see it as invalidated.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby";
+
+# Synced slot mustn't get invalidated on the standby even after a checkpoint,
+# it must sync invalidation from the primary. So, we must not see the slot's
+# invalidation message in server log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned slot lsub1_sync_slot has not been invalidated on the standby'
+);
+
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate streaming standby's slot as well as logical failover
+# slot on primary due to replication_slot_inactive_timeout. Also, check the
+# logical failover slot synced on to the standby doesn't invalidate the slot on
+# its own, but gets the invalidated state from the remote slot on the primary.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+
+my $publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber node
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
+
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for slot $slot_name to become inactive on node $name";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of slot $slot_name to be updated on node $name";
+
+ # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
+ # for the slot to get invalidated.
+ sleep($inactive_timeout);
+
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
+
+ # Check that the invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot_name', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot_name"/,
+ "detected error upon trying to acquire invalidated slot $slot_name on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot_name on node $name";
+}
+
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $name = $node->name;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot_name\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot_name invalidation has been logged on node $name"
+ );
+}
+
+done_testing();
--
2.43.0
[application/x-patch] v42-0002-Add-XID-age-based-replication-slot-invalidation.patch (32.4K, ../../CALj2ACVu0EJ+QQ7W0oHbL_+hCtk+p-00MNhE4hfjFcXaMepMfg@mail.gmail.com/3-v42-0002-Add-XID-age-based-replication-slot-invalidation.patch)
download | inline diff:
From 4106d3edcb3b5b37eb198bfb076863cb99300813 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Mon, 26 Aug 2024 05:36:34 +0000
Subject: [PATCH v42 2/2] Add XID age based replication slot invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set an XID age (age of
slot's xmin or catalog_xmin) of say 1 or 1.5 billion, after which
the slots get invalidated.
To achieve the above, postgres introduces a GUC allowing users
set slot XID age. The replication slots whose xmin or catalog_xmin
has reached the age specified by this setting get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
- During vacuum (both command-based and autovacuum)
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/20240327150557.GA3994937%40nathanxps13
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoaRECcnyqxAxUhP5dk2S4HX%3DpGh-p-PkA3uc%2BjG_9hiMw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 26 ++
doc/src/sgml/system-views.sgml | 8 +
src/backend/commands/vacuum.c | 66 ++++
src/backend/replication/slot.c | 158 ++++++++-
src/backend/utils/misc/guc_tables.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 3 +
src/test/recovery/t/050_invalidate_slots.pl | 321 +++++++++++++++++-
8 files changed, 574 insertions(+), 19 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index fbbacbee5b..61505ed6aa 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4593,6 +4593,32 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-xid-age" xreflabel="replication_slot_xid_age">
+ <term><varname>replication_slot_xid_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_xid_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots whose <literal>xmin</literal> (the oldest
+ transaction that this slot needs the database to retain) or
+ <literal>catalog_xmin</literal> (the oldest transaction affecting the
+ system catalogs that this slot needs the database to retain) has reached
+ the age specified by this setting. A value of zero (which is default)
+ disables this feature. Users can set this value anywhere from zero to
+ two billion. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command
+ line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during vacuum or during checkpoint.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 9e00f7d184..a4f1ab5275 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2625,6 +2625,14 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>xid_aged</literal> means that the slot's
+ <literal>xmin</literal> or <literal>catalog_xmin</literal>
+ has reached the age specified by
+ <xref linkend="guc-replication-slot-xid-age"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 7d8e9d2045..c909c0d001 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -47,6 +47,7 @@
#include "postmaster/autovacuum.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/interrupt.h"
+#include "replication/slot.h"
#include "storage/bufmgr.h"
#include "storage/lmgr.h"
#include "storage/pmsignal.h"
@@ -116,6 +117,7 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
+static void try_replication_slot_invalidation(void);
/*
* GUC check function to ensure GUC value specified is within the allowable
@@ -452,6 +454,61 @@ ExecVacuum(ParseState *pstate, VacuumStmt *vacstmt, bool isTopLevel)
MemoryContextDelete(vac_context);
}
+/*
+ * Try invalidating replication slots based on current replication slot xmin
+ * limits once every vacuum cycle.
+ */
+static void
+try_replication_slot_invalidation(void)
+{
+ TransactionId min_slot_xmin;
+ TransactionId min_slot_catalog_xmin;
+ bool can_invalidate = false;
+ TransactionId cutoff;
+ TransactionId curr;
+
+ curr = ReadNextTransactionId();
+
+ /*
+ * The cutoff can tell how far we can go back from the current transaction
+ * id till the age. And then, we check whether or not the xmin or
+ * catalog_xmin falls within the cutoff; if yes, return true, otherwise
+ * false.
+ */
+ cutoff = curr - replication_slot_xid_age;
+
+ if (!TransactionIdIsNormal(cutoff))
+ cutoff = FirstNormalTransactionId;
+
+ ProcArrayGetReplicationSlotXmin(&min_slot_xmin, &min_slot_catalog_xmin);
+
+ /*
+ * Current replication slot xmin limits can never be larger than the
+ * current transaction id even in the case of transaction ID wraparound.
+ */
+ Assert(min_slot_xmin <= curr);
+ Assert(min_slot_catalog_xmin <= curr);
+
+ if (TransactionIdIsNormal(min_slot_xmin) &&
+ TransactionIdPrecedesOrEquals(min_slot_xmin, cutoff))
+ can_invalidate = true;
+ else if (TransactionIdIsNormal(min_slot_catalog_xmin) &&
+ TransactionIdPrecedesOrEquals(min_slot_catalog_xmin, cutoff))
+ can_invalidate = true;
+
+ if (can_invalidate)
+ {
+ /*
+ * Note that InvalidateObsoleteReplicationSlots is also called as part
+ * of CHECKPOINT, and emitting ERRORs from within is avoided already.
+ * Therefore, there is no concern here that any ERROR from
+ * invalidating replication slots blocks VACUUM.
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_XID_AGE, 0,
+ InvalidOid, InvalidTransactionId);
+ }
+}
+
/*
* Internal entry point for autovacuum and the VACUUM / ANALYZE commands.
*
@@ -483,6 +540,7 @@ vacuum(List *relations, VacuumParams *params, BufferAccessStrategy bstrategy,
const char *stmttype;
volatile bool in_outer_xact,
use_own_xacts;
+ static bool first_time = true;
Assert(params != NULL);
@@ -594,6 +652,14 @@ vacuum(List *relations, VacuumParams *params, BufferAccessStrategy bstrategy,
CommitTransactionCommand();
}
+ if (params->options & VACOPT_VACUUM &&
+ first_time &&
+ replication_slot_xid_age > 0)
+ {
+ try_replication_slot_invalidation();
+ first_time = false;
+ }
+
/* Turn vacuum cost accounting on or off, and set/clear in_vacuum */
PG_TRY();
{
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 5f504f956f..0654a81add 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -108,10 +108,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
[RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
+ [RS_INVAL_XID_AGE] = "xid_aged",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
+#define RS_INVAL_MAX_CAUSES RS_INVAL_XID_AGE
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -142,6 +143,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
int max_replication_slots = 10; /* the maximum number of replication
* slots */
int replication_slot_inactive_timeout = 0;
+int replication_slot_xid_age = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -160,6 +162,9 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool ReplicationSlotIsXIDAged(ReplicationSlot *slot,
+ TransactionId *xmin,
+ TransactionId *catalog_xmin);
static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReplicationSlot *s,
@@ -636,8 +641,8 @@ retry:
* it gets invalidated now or has been invalidated previously, because
* there's no use in acquiring the invalidated slot.
*
- * XXX: Currently we check for inactive_timeout invalidation here. We
- * might need to check for other invalidations too.
+ * XXX: Currently we check for inactive_timeout and xid_aged invalidations
+ * here. We might need to check for other invalidations too.
*/
if (check_for_invalidation)
{
@@ -648,6 +653,22 @@ retry:
InvalidTransactionId,
&invalidated);
+ if (!invalidated && released_lock)
+ {
+ /* The slot is still ours */
+ Assert(s->active_pid == MyProcPid);
+
+ /* Reacquire the ControlLock */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ released_lock = false;
+ }
+
+ if (!invalidated)
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_XID_AGE,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ &invalidated);
+
/*
* If the slot has been invalidated, recalculate the resource limits.
*/
@@ -657,7 +678,8 @@ retry:
ReplicationSlotsComputeRequiredLSN();
}
- if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT ||
+ s->data.invalidated == RS_INVAL_XID_AGE)
{
/*
* Release the lock if it hasn't been already, to ensure smooth
@@ -665,7 +687,10 @@ retry:
*/
if (!released_lock)
LWLockRelease(ReplicationSlotControlLock);
+ }
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
Assert(s->inactive_since > 0);
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -675,6 +700,20 @@ retry:
timestamptz_to_str(s->inactive_since),
replication_slot_inactive_timeout)));
}
+
+ if (s->data.invalidated == RS_INVAL_XID_AGE)
+ {
+ Assert(TransactionIdIsValid(s->data.xmin) ||
+ TransactionIdIsValid(s->data.catalog_xmin));
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("The slot's xmin %u or catalog_xmin %u has reached the age %d specified by \"replication_slot_xid_age\".",
+ s->data.xmin,
+ s->data.catalog_xmin,
+ replication_slot_xid_age)));
+ }
}
if (!released_lock)
@@ -1567,7 +1606,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
TransactionId snapshotConflictHorizon,
- TimestampTz inactive_since)
+ TimestampTz inactive_since,
+ TransactionId xmin,
+ TransactionId catalog_xmin)
{
StringInfoData err_detail;
bool hint = false;
@@ -1604,6 +1645,20 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
timestamptz_to_str(inactive_since),
replication_slot_inactive_timeout);
break;
+ case RS_INVAL_XID_AGE:
+ Assert(TransactionIdIsValid(xmin) ||
+ TransactionIdIsValid(catalog_xmin));
+
+ if (TransactionIdIsValid(xmin))
+ appendStringInfo(&err_detail, _("The slot's xmin %u has reached the age %d specified by \"replication_slot_xid_age\"."),
+ xmin,
+ replication_slot_xid_age);
+ else if (TransactionIdIsValid(catalog_xmin))
+ appendStringInfo(&err_detail, _("The slot's catalog_xmin %u has reached the age %d specified by \"replication_slot_xid_age\"."),
+ catalog_xmin,
+ replication_slot_xid_age);
+
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1648,6 +1703,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
TimestampTz inactive_since = 0;
+ TransactionId aged_xmin = InvalidTransactionId;
+ TransactionId aged_catalog_xmin = InvalidTransactionId;
for (;;)
{
@@ -1764,6 +1821,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
Assert(s->active_pid == 0);
}
break;
+ case RS_INVAL_XID_AGE:
+ if (ReplicationSlotIsXIDAged(s, &aged_xmin, &aged_catalog_xmin))
+ {
+ Assert(TransactionIdIsValid(aged_xmin) ||
+ TransactionIdIsValid(aged_catalog_xmin));
+
+ invalidation_cause = cause;
+ break;
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1852,7 +1919,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
oldestLSN, snapshotConflictHorizon,
- inactive_since);
+ inactive_since, aged_xmin,
+ aged_catalog_xmin);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1899,7 +1967,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
oldestLSN, snapshotConflictHorizon,
- inactive_since);
+ inactive_since, aged_xmin,
+ aged_catalog_xmin);
/* done with this slot for now */
break;
@@ -1923,6 +1992,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
* - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
+ * - RS_INVAL_XID_AGE: slot's xmin or catalog_xmin has reached the age
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1986,6 +2056,7 @@ void
CheckPointReplicationSlots(bool is_shutdown)
{
int i;
+ bool invalidated;
elog(DEBUG1, "performing replication slot checkpoint");
@@ -2053,11 +2124,76 @@ CheckPointReplicationSlots(bool is_shutdown)
*
* - Avoid saving slot info to disk two times for each invalidated slot.
*
- * XXX: Should we move inactive_timeout inavalidation check closer to
- * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ * XXX: Should we move inactive_timeout and xid_aged inavalidation checks
+ * closer to wal_removed in CreateCheckPoint and CreateRestartPoint?
*/
- InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
- 0, InvalidOid, InvalidTransactionId);
+ invalidated = InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+
+ if (!invalidated)
+ InvalidateObsoleteReplicationSlots(RS_INVAL_XID_AGE,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+}
+
+/*
+ * Returns true if the given replication slot's xmin or catalog_xmin age is
+ * more than replication_slot_xid_age.
+ *
+ * Note that the caller must hold the replication slot's spinlock to avoid
+ * race conditions while this function reads xmin and catalog_xmin.
+ */
+static bool
+ReplicationSlotIsXIDAged(ReplicationSlot *slot, TransactionId *xmin,
+ TransactionId *catalog_xmin)
+{
+ TransactionId cutoff;
+ TransactionId curr;
+
+ if (replication_slot_xid_age == 0)
+ return false;
+
+ curr = ReadNextTransactionId();
+
+ /*
+ * Replication slot's xmin and catalog_xmin can never be larger than the
+ * current transaction id even in the case of transaction ID wraparound.
+ */
+ Assert(slot->data.xmin <= curr);
+ Assert(slot->data.catalog_xmin <= curr);
+
+ /*
+ * The cutoff can tell how far we can go back from the current transaction
+ * id till the age. And then, we check whether or not the xmin or
+ * catalog_xmin falls within the cutoff; if yes, return true, otherwise
+ * false.
+ */
+ cutoff = curr - replication_slot_xid_age;
+
+ if (!TransactionIdIsNormal(cutoff))
+ cutoff = FirstNormalTransactionId;
+
+ *xmin = InvalidTransactionId;
+ *catalog_xmin = InvalidTransactionId;
+
+ if (TransactionIdIsNormal(slot->data.xmin) &&
+ TransactionIdPrecedesOrEquals(slot->data.xmin, cutoff))
+ {
+ *xmin = slot->data.xmin;
+ return true;
+ }
+
+ if (TransactionIdIsNormal(slot->data.catalog_xmin) &&
+ TransactionIdPrecedesOrEquals(slot->data.catalog_xmin, cutoff))
+ {
+ *catalog_xmin = slot->data.catalog_xmin;
+ return true;
+ }
+
+ return false;
}
/*
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 861692c683..5d10dd1c8a 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3040,6 +3040,16 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_xid_age", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Age of the transaction ID at which a replication slot gets invalidated."),
+ gettext_noop("The transaction is the oldest transaction (including the one affecting the system catalogs) that a replication slot needs the database to retain.")
+ },
+ &replication_slot_xid_age,
+ 0, 0, 2000000000,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index deca3a4aeb..3fb6813195 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
+#replication_slot_xid_age = 0
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index dd56a77547..5ea73f9331 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -55,6 +55,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_WAL_LEVEL,
/* inactive slot timeout has occurred */
RS_INVAL_INACTIVE_TIMEOUT,
+ /* slot's xmin or catalog_xmin has reached the age */
+ RS_INVAL_XID_AGE,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -233,6 +235,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
extern PGDLLIMPORT int replication_slot_inactive_timeout;
+extern PGDLLIMPORT int replication_slot_xid_age;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
index 4663019c16..18300cfeca 100644
--- a/src/test/recovery/t/050_invalidate_slots.pl
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -89,7 +89,7 @@ $primary->reload;
# that nobody has acquired that slot yet, so due to
# replication_slot_inactive_timeout setting above it must get invalidated.
wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Set timeout on the standby also to check the synced slots don't get
# invalidated due to timeout on the standby.
@@ -129,7 +129,7 @@ $standby1->stop;
# Wait for the standby's replication slot to become inactive
wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Testcase end: Invalidate streaming standby's slot as well as logical failover
# slot on primary due to replication_slot_inactive_timeout. Also, check the
@@ -197,15 +197,280 @@ $subscriber->stop;
# Wait for the replication slot to become inactive and then invalidated due to
# timeout.
wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Testcase end: Invalidate logical subscriber's slot due to
# replication_slot_inactive_timeout.
# =============================================================================
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot due to replication_slot_xid_age
+# GUC.
+
+# Prepare for the next test
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$primary->reload;
+
+# Create a standby linking to the primary using the replication slot
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+# Enable hs_feedback. The slot should gain an xmin. We set the status interval
+# so we'll see the results promptly.
+$standby2->append_conf(
+ 'postgresql.conf', q{
+primary_slot_name = 'sb2_slot'
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+});
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb2_slot', immediately_reserve := true);
+]);
+
+$standby2->start;
+
+# Create some content on primary to move xmin
+$primary->safe_psql('postgres',
+ "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby2);
+
+$primary->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NOT NULL AND catalog_xmin IS NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'sb2_slot';
+]) or die "Timed out waiting for slot sb2_slot xmin to advance";
+
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$primary->reload;
+
+# Stop standby to make the replication slot's xmin on primary to age
+$standby2->stop;
+
+$logstart = -s $primary->logfile;
+
+# Do some work to advance xids on primary
+advance_xids($primary, 'tab_int');
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($primary, 'sb2_slot', $logstart, 0, 'xid_aged');
+
+# Testcase end: Invalidate streaming standby's slot due to replication_slot_xid_age
+# GUC.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_xid_age GUC.
+
+$publisher = $primary;
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$publisher->reload;
+
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+));
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl2 (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl2 (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl2 VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+$publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE test_tbl2");
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (slot_name = 'lsub2_slot')"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub2');
+
+$result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl2");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NULL AND catalog_xmin IS NOT NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'lsub2_slot';
+]) or die "Timed out waiting for slot lsub2_slot catalog_xmin to advance";
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Do some work to advance xids on publisher
+advance_xids($publisher, 'test_tbl2');
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($publisher, 'lsub2_slot', $logstart, 0,
+ 'xid_aged');
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_xid_age GUC.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical slot on standby that's being synced from
+# the primary due to replication_slot_xid_age GUC.
+
+$publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 0;
+]);
+$publisher->reload;
+
+# Create a standby linking to the primary using the replication slot
+my $standby3 = PostgreSQL::Test::Cluster->new('standby3');
+$standby3->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+$standby3->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb3_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb3_slot', immediately_reserve := true);
+]);
+
+$standby3->start;
+
+my $standby3_logstart = -s $standby3->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby3);
+
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+));
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl3 (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl3 (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl3 VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+$publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE test_tbl3");
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (slot_name = 'lsub3_sync_slot', failover = true)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub3');
+
+$result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl3");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NULL AND catalog_xmin IS NOT NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'lsub3_sync_slot';
+])
+ or die "Timed out waiting for slot lsub3_sync_slot catalog_xmin to advance";
+
+# Synchronize the primary server slots to the standby
+$standby3->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced' and has got catalog_xmin from the primary.
+is( $standby3->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub3_sync_slot' AND synced AND NOT temporary AND
+ xmin IS NULL AND catalog_xmin IS NOT NULL;}
+ ),
+ "t",
+ 'logical slot has synced as true on standby');
+
+my $primary_catalog_xmin = $primary->safe_psql('postgres',
+ "SELECT catalog_xmin FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND catalog_xmin IS NOT NULL;"
+);
+
+my $stabdby3_catalog_xmin = $standby3->safe_psql('postgres',
+ "SELECT catalog_xmin FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND catalog_xmin IS NOT NULL;"
+);
+
+is($primary_catalog_xmin, $stabdby3_catalog_xmin,
+ "check catalog_xmin are same for primary slot and synced slot");
+
+# Enable XID age based invalidation on the standby. Note that we disabled the
+# same on the primary to check if the invalidation occurs for synced slot on
+# the standby.
+$standby3->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$standby3->reload;
+
+$logstart = -s $standby3->logfile;
+
+# Do some work to advance xids on primary
+advance_xids($primary, 'test_tbl3');
+
+# Wait for standby to catch up with the above work
+$primary->wait_for_catchup($standby3);
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($standby3, 'lsub3_sync_slot', $logstart, 0,
+ 'xid_aged');
+
+# Note that the replication slot on the primary is still active
+$result = $primary->safe_psql('postgres',
+ "SELECT COUNT(slot_name) = 1 FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND invalidation_reason IS NULL;"
+);
+
+is($result, 't', "check lsub3_sync_slot is still active on primary");
+
+# Testcase end: Invalidate logical slot on standby that's being synced from
+# the primary due to replication_slot_xid_age GUC.
+# =============================================================================
+
sub wait_for_slot_invalidation
{
- my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my ($node, $slot_name, $offset, $inactive_timeout, $reason) = @_;
my $name = $node->name;
# Wait for the replication slot to become inactive
@@ -231,14 +496,15 @@ sub wait_for_slot_invalidation
# for the slot to get invalidated.
sleep($inactive_timeout);
- check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset,
+ $reason);
# Wait for the inactive replication slot to be invalidated
$node->poll_query_until(
'postgres', qq[
SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
WHERE slot_name = '$slot_name' AND
- invalidation_reason = 'inactive_timeout';
+ invalidation_reason = '$reason';
])
or die
"Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
@@ -262,15 +528,33 @@ sub wait_for_slot_invalidation
# Check for invalidation of slot in server log
sub check_for_slot_invalidation_in_server_log
{
- my ($node, $slot_name, $offset) = @_;
+ my ($node, $slot_name, $offset, $reason) = @_;
my $name = $node->name;
my $invalidated = 0;
+ my $isrecovery =
+ $node->safe_psql('postgres', "SELECT pg_is_in_recovery()");
+
+ chomp($isrecovery);
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
- $node->safe_psql('postgres', "CHECKPOINT");
+ if ($reason eq 'xid_aged' && $isrecovery eq 'f')
+ {
+ $node->safe_psql('postgres', "VACUUM");
+ }
+ else
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ }
+
if ($node->log_contains(
"invalidating obsolete replication slot \"$slot_name\"",
+ $offset)
+ || $node->log_contains(
+ "The slot's xmin .* has reached the age .* specified by \"replication_slot_xid_age\".",
+ $offset)
+ || $node->log_contains(
+ "The slot's catalog_xmin .* has reached the age .* specified by \"replication_slot_xid_age\".",
$offset))
{
$invalidated = 1;
@@ -283,4 +567,25 @@ sub check_for_slot_invalidation_in_server_log
);
}
+# Do some work for advancing xids on a given node
+sub advance_xids
+{
+ my ($node, $table_name) = @_;
+
+ $node->safe_psql(
+ 'postgres', qq[
+ do \$\$
+ begin
+ for i in 10000..11000 loop
+ -- use an exception block so that each iteration eats an XID
+ begin
+ insert into $table_name values (i);
+ exception
+ when division_by_zero then null;
+ end;
+ end loop;
+ end\$\$;
+ ]);
+}
+
done_testing();
--
2.43.0
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-08-26 11:05 ` Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Amit Kapila @ 2024-08-26 11:05 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Aug 26, 2024 at 11:44 AM Bharath Rupireddy
<[email protected]> wrote:
>
Few comments on 0001:
1.
@@ -651,6 +651,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid
remote_dbid)
" name slot \"%s\" already exists on the standby",
remote_slot->name));
+ /*
+ * Skip the sync if the local slot is already invalidated. We do this
+ * beforehand to avoid slot acquire and release.
+ */
+ if (slot->data.invalidated != RS_INVAL_NONE)
+ return false;
+
/*
* The slot has been synchronized before.
I was wondering why you have added this new check as part of this
patch. If you see the following comments in the related code, you will
know why we haven't done this previously.
/*
* The slot has been synchronized before.
*
* It is important to acquire the slot here before checking
* invalidation. If we don't acquire the slot first, there could be a
* race condition that the local slot could be invalidated just after
* checking the 'invalidated' flag here and we could end up
* overwriting 'invalidated' flag to remote_slot's value. See
* InvalidatePossiblyObsoleteSlot() where it invalidates slot directly
* if the slot is not acquired by other processes.
*
* XXX: If it ever turns out that slot acquire/release is costly for
* cases when none of the slot properties is changed then we can do a
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
ReplicationSlotAcquire(remote_slot->name, true);
We need some modifications in these comments if you want to add a
pre-check here.
2.
@@ -1907,6 +2033,31 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * Note that we will make another pass over replication slots for
+ * invalidations to keep the code simple. The assumption here is that the
+ * traversal over replication slots isn't that costly even with hundreds
+ * of replication slots. If it ever turns out that this assumption is
+ * wrong, we might have to put the invalidation check logic in the above
+ * loop, for that we might have to do the following:
+ *
+ * - Acqure ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated slot.
+ *
+ * XXX: Should we move inactive_timeout inavalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0, InvalidOid, InvalidTransactionId);
Why do we want to call this for shutdown case (when is_shutdown is
true)? I understand trying to invalidate slots during regular
checkpoint but not sure if we need it at the time of shutdown. The
other point is can we try to check the performance impact with 100s of
slots as mentioned in the code comments?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
@ 2024-08-29 06:01 ` Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-02 06:55 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
0 siblings, 2 replies; 98+ messages in thread
From: Bharath Rupireddy @ 2024-08-29 06:01 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Thanks for looking into this.
On Mon, Aug 26, 2024 at 4:35 PM Amit Kapila <[email protected]> wrote:
>
> Few comments on 0001:
> 1.
> @@ -651,6 +651,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid
>
> + /*
> + * Skip the sync if the local slot is already invalidated. We do this
> + * beforehand to avoid slot acquire and release.
> + */
>
> I was wondering why you have added this new check as part of this
> patch. If you see the following comments in the related code, you will
> know why we haven't done this previously.
Removed. Can deal with optimization separately.
> 2.
> + */
> + InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
> + 0, InvalidOid, InvalidTransactionId);
>
> Why do we want to call this for shutdown case (when is_shutdown is
> true)? I understand trying to invalidate slots during regular
> checkpoint but not sure if we need it at the time of shutdown.
Changed it to invalidate only for non-shutdown checkpoints.
inactive_timeout invalidation isn't critical for shutdown unlike
wal_removed which can help shutdown by freeing up some disk space.
> The
> other point is can we try to check the performance impact with 100s of
> slots as mentioned in the code comments?
I first checked how much does the wal_removed invalidation check add to the
checkpoint (see 2nd and 3rd column). I then checked how much
inactive_timeout invalidation check adds to the checkpoint (see 4th
column), it is not more than wal_remove invalidation check. I then checked
how much the wal_removed invalidation check adds for replication slots that
have already been invalidated due to inactive_timeout (see 5th column),
looks like not much.
| # of slots | HEAD (no invalidation) ms | HEAD (wal_removed) ms | PATCHED
(inactive_timeout) ms | PATCHED (inactive_timeout+wal_removed) ms |
|------------|----------------------------|-----------------------|-------------------------------|------------------------------------------|
| 100 | 18.591 | 370.586 | 359.299
| 373.882 |
| 1000 | 15.722 | 4834.901 |
5081.751 | 5072.128 |
| 10000 | 19.261 | 59801.062 |
61270.406 | 60270.099 |
Having said that, I'm okay to implement the optimization specified.
Thoughts?
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
Please see the attached v43 patches addressing the above review comments.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v43-0002-Add-XID-age-based-replication-slot-invalidation.patch (33.0K, ../../CALj2ACXe8+xSNdMXTMaSRWUwX7v61Ad4iddUwnn=djSwx3GLLg@mail.gmail.com/3-v43-0002-Add-XID-age-based-replication-slot-invalidation.patch)
download | inline diff:
From bb28e7ba7ba783b0c908412774b1d6ea4cca6dc5 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Thu, 29 Aug 2024 05:11:31 +0000
Subject: [PATCH v43 2/2] Add XID age based replication slot invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set an XID age (age of
slot's xmin or catalog_xmin) of say 1 or 1.5 billion, after which
the slots get invalidated.
To achieve the above, postgres introduces a GUC allowing users
set slot XID age. The replication slots whose xmin or catalog_xmin
has reached the age specified by this setting get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
The invalidation check happens at various locations to help beingas latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
- During vacuum (both command-based and autovacuum)
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/20240327150557.GA3994937%40nathanxps13
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoaRECcnyqxAxUhP5dk2S4HX%3DpGh-p-PkA3uc%2BjG_9hiMw%40mail.gmail.com
---
doc/src/sgml/config.sgml | 26 ++
doc/src/sgml/system-views.sgml | 8 +
src/backend/commands/vacuum.c | 66 ++++
src/backend/replication/slot.c | 157 ++++++++-
src/backend/utils/misc/guc_tables.c | 10 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 3 +
src/test/recovery/t/050_invalidate_slots.pl | 321 +++++++++++++++++-
8 files changed, 572 insertions(+), 20 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 113303a501..a65c49d9fa 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4589,6 +4589,32 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-xid-age" xreflabel="replication_slot_xid_age">
+ <term><varname>replication_slot_xid_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_xid_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots whose <literal>xmin</literal> (the oldest
+ transaction that this slot needs the database to retain) or
+ <literal>catalog_xmin</literal> (the oldest transaction affecting the
+ system catalogs that this slot needs the database to retain) has reached
+ the age specified by this setting. A value of zero (which is default)
+ disables this feature. Users can set this value anywhere from zero to
+ two billion. This parameter can only be set in the
+ <filename>postgresql.conf</filename> file or on the server command
+ line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during vacuum or during checkpoint.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 9e00f7d184..a4f1ab5275 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2625,6 +2625,14 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>xid_aged</literal> means that the slot's
+ <literal>xmin</literal> or <literal>catalog_xmin</literal>
+ has reached the age specified by
+ <xref linkend="guc-replication-slot-xid-age"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/commands/vacuum.c b/src/backend/commands/vacuum.c
index 7d8e9d2045..c909c0d001 100644
--- a/src/backend/commands/vacuum.c
+++ b/src/backend/commands/vacuum.c
@@ -47,6 +47,7 @@
#include "postmaster/autovacuum.h"
#include "postmaster/bgworker_internals.h"
#include "postmaster/interrupt.h"
+#include "replication/slot.h"
#include "storage/bufmgr.h"
#include "storage/lmgr.h"
#include "storage/pmsignal.h"
@@ -116,6 +117,7 @@ static bool vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params,
static double compute_parallel_delay(void);
static VacOptValue get_vacoptval_from_boolean(DefElem *def);
static bool vac_tid_reaped(ItemPointer itemptr, void *state);
+static void try_replication_slot_invalidation(void);
/*
* GUC check function to ensure GUC value specified is within the allowable
@@ -452,6 +454,61 @@ ExecVacuum(ParseState *pstate, VacuumStmt *vacstmt, bool isTopLevel)
MemoryContextDelete(vac_context);
}
+/*
+ * Try invalidating replication slots based on current replication slot xmin
+ * limits once every vacuum cycle.
+ */
+static void
+try_replication_slot_invalidation(void)
+{
+ TransactionId min_slot_xmin;
+ TransactionId min_slot_catalog_xmin;
+ bool can_invalidate = false;
+ TransactionId cutoff;
+ TransactionId curr;
+
+ curr = ReadNextTransactionId();
+
+ /*
+ * The cutoff can tell how far we can go back from the current transaction
+ * id till the age. And then, we check whether or not the xmin or
+ * catalog_xmin falls within the cutoff; if yes, return true, otherwise
+ * false.
+ */
+ cutoff = curr - replication_slot_xid_age;
+
+ if (!TransactionIdIsNormal(cutoff))
+ cutoff = FirstNormalTransactionId;
+
+ ProcArrayGetReplicationSlotXmin(&min_slot_xmin, &min_slot_catalog_xmin);
+
+ /*
+ * Current replication slot xmin limits can never be larger than the
+ * current transaction id even in the case of transaction ID wraparound.
+ */
+ Assert(min_slot_xmin <= curr);
+ Assert(min_slot_catalog_xmin <= curr);
+
+ if (TransactionIdIsNormal(min_slot_xmin) &&
+ TransactionIdPrecedesOrEquals(min_slot_xmin, cutoff))
+ can_invalidate = true;
+ else if (TransactionIdIsNormal(min_slot_catalog_xmin) &&
+ TransactionIdPrecedesOrEquals(min_slot_catalog_xmin, cutoff))
+ can_invalidate = true;
+
+ if (can_invalidate)
+ {
+ /*
+ * Note that InvalidateObsoleteReplicationSlots is also called as part
+ * of CHECKPOINT, and emitting ERRORs from within is avoided already.
+ * Therefore, there is no concern here that any ERROR from
+ * invalidating replication slots blocks VACUUM.
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_XID_AGE, 0,
+ InvalidOid, InvalidTransactionId);
+ }
+}
+
/*
* Internal entry point for autovacuum and the VACUUM / ANALYZE commands.
*
@@ -483,6 +540,7 @@ vacuum(List *relations, VacuumParams *params, BufferAccessStrategy bstrategy,
const char *stmttype;
volatile bool in_outer_xact,
use_own_xacts;
+ static bool first_time = true;
Assert(params != NULL);
@@ -594,6 +652,14 @@ vacuum(List *relations, VacuumParams *params, BufferAccessStrategy bstrategy,
CommitTransactionCommand();
}
+ if (params->options & VACOPT_VACUUM &&
+ first_time &&
+ replication_slot_xid_age > 0)
+ {
+ try_replication_slot_invalidation();
+ first_time = false;
+ }
+
/* Turn vacuum cost accounting on or off, and set/clear in_vacuum */
PG_TRY();
{
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 70093500fa..530c121c2f 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -108,10 +108,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
[RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
+ [RS_INVAL_XID_AGE] = "xid_aged",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
+#define RS_INVAL_MAX_CAUSES RS_INVAL_XID_AGE
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -142,6 +143,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
int max_replication_slots = 10; /* the maximum number of replication
* slots */
int replication_slot_inactive_timeout = 0;
+int replication_slot_xid_age = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -160,6 +162,9 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool ReplicationSlotIsXIDAged(ReplicationSlot *slot,
+ TransactionId *xmin,
+ TransactionId *catalog_xmin);
static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReplicationSlot *s,
@@ -636,8 +641,8 @@ retry:
* gets invalidated now or has been invalidated previously, because
* there's no use in acquiring the invalidated slot.
*
- * XXX: Currently we check for inactive_timeout invalidation here. We
- * might need to check for other invalidations too.
+ * XXX: Currently we check for inactive_timeout and xid_aged invalidations
+ * here. We might need to check for other invalidations too.
*/
if (check_for_invalidation)
{
@@ -648,6 +653,22 @@ retry:
InvalidTransactionId,
&invalidated);
+ if (!invalidated && released_lock)
+ {
+ /* The slot is still ours */
+ Assert(s->active_pid == MyProcPid);
+
+ /* Reacquire the ControlLock */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ released_lock = false;
+ }
+
+ if (!invalidated)
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_XID_AGE,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ &invalidated);
+
/*
* If the slot has been invalidated, recalculate the resource limits.
*/
@@ -657,7 +678,8 @@ retry:
ReplicationSlotsComputeRequiredLSN();
}
- if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT ||
+ s->data.invalidated == RS_INVAL_XID_AGE)
{
/*
* Release the lock if it hasn't been already, to ensure smooth
@@ -665,7 +687,10 @@ retry:
*/
if (!released_lock)
LWLockRelease(ReplicationSlotControlLock);
+ }
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
Assert(s->inactive_since > 0);
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -675,6 +700,20 @@ retry:
timestamptz_to_str(s->inactive_since),
replication_slot_inactive_timeout)));
}
+
+ if (s->data.invalidated == RS_INVAL_XID_AGE)
+ {
+ Assert(TransactionIdIsValid(s->data.xmin) ||
+ TransactionIdIsValid(s->data.catalog_xmin));
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("The slot's xmin %u or catalog_xmin %u has reached the age %d specified by \"replication_slot_xid_age\".",
+ s->data.xmin,
+ s->data.catalog_xmin,
+ replication_slot_xid_age)));
+ }
}
if (!released_lock)
@@ -1567,7 +1606,9 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
TransactionId snapshotConflictHorizon,
- TimestampTz inactive_since)
+ TimestampTz inactive_since,
+ TransactionId xmin,
+ TransactionId catalog_xmin)
{
StringInfoData err_detail;
bool hint = false;
@@ -1604,6 +1645,20 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
timestamptz_to_str(inactive_since),
replication_slot_inactive_timeout);
break;
+ case RS_INVAL_XID_AGE:
+ Assert(TransactionIdIsValid(xmin) ||
+ TransactionIdIsValid(catalog_xmin));
+
+ if (TransactionIdIsValid(xmin))
+ appendStringInfo(&err_detail, _("The slot's xmin %u has reached the age %d specified by \"replication_slot_xid_age\"."),
+ xmin,
+ replication_slot_xid_age);
+ else if (TransactionIdIsValid(catalog_xmin))
+ appendStringInfo(&err_detail, _("The slot's catalog_xmin %u has reached the age %d specified by \"replication_slot_xid_age\"."),
+ catalog_xmin,
+ replication_slot_xid_age);
+
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1648,6 +1703,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
TimestampTz inactive_since = 0;
+ TransactionId aged_xmin = InvalidTransactionId;
+ TransactionId aged_catalog_xmin = InvalidTransactionId;
for (;;)
{
@@ -1764,6 +1821,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
Assert(s->active_pid == 0);
}
break;
+ case RS_INVAL_XID_AGE:
+ if (ReplicationSlotIsXIDAged(s, &aged_xmin, &aged_catalog_xmin))
+ {
+ Assert(TransactionIdIsValid(aged_xmin) ||
+ TransactionIdIsValid(aged_catalog_xmin));
+
+ invalidation_cause = cause;
+ break;
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1852,7 +1919,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
oldestLSN, snapshotConflictHorizon,
- inactive_since);
+ inactive_since, aged_xmin,
+ aged_catalog_xmin);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1899,7 +1967,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
oldestLSN, snapshotConflictHorizon,
- inactive_since);
+ inactive_since, aged_xmin,
+ aged_catalog_xmin);
/* done with this slot for now */
break;
@@ -1923,6 +1992,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
* - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
+ * - RS_INVAL_XID_AGE: slot's xmin or catalog_xmin has reached the age
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -2043,10 +2113,11 @@ CheckPointReplicationSlots(bool is_shutdown)
* NB: We will make another pass over replication slots for
* invalidation checks to keep the code simple. Testing shows that
* there is no noticeable overhead (when compared with wal_removed
- * invalidation) even if we were to do inactive_timeout invalidation
- * of thousands of replication slots here. If it is ever proven that
- * this assumption is wrong, we will have to perform the invalidation
- * checks in the above for loop with the following changes:
+ * invalidation) even if we were to do inactive_timeout/xid_aged
+ * invalidation of thousands of replication slots here. If it is ever
+ * proven that this assumption is wrong, we will have to perform the
+ * invalidation checks in the above for loop with the following
+ * changes:
*
* - Acquire ControlLock lock once before the loop.
*
@@ -2058,16 +2129,78 @@ CheckPointReplicationSlots(bool is_shutdown)
* - Avoid saving slot info to disk two times for each invalidated
* slot.
*
- * XXX: Should we move inactive_timeout inavalidation check closer to
+ * XXX: Should we move these inavalidation checks closer to
* wal_removed in CreateCheckPoint and CreateRestartPoint?
*/
InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
0,
InvalidOid,
InvalidTransactionId);
+
+ InvalidateObsoleteReplicationSlots(RS_INVAL_XID_AGE,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
}
}
+/*
+ * Returns true if the given replication slot's xmin or catalog_xmin age is
+ * more than replication_slot_xid_age.
+ *
+ * Note that the caller must hold the replication slot's spinlock to avoid
+ * race conditions while this function reads xmin and catalog_xmin.
+ */
+static bool
+ReplicationSlotIsXIDAged(ReplicationSlot *slot, TransactionId *xmin,
+ TransactionId *catalog_xmin)
+{
+ TransactionId cutoff;
+ TransactionId curr;
+
+ if (replication_slot_xid_age == 0)
+ return false;
+
+ curr = ReadNextTransactionId();
+
+ /*
+ * Replication slot's xmin and catalog_xmin can never be larger than the
+ * current transaction id even in the case of transaction ID wraparound.
+ */
+ Assert(slot->data.xmin <= curr);
+ Assert(slot->data.catalog_xmin <= curr);
+
+ /*
+ * The cutoff can tell how far we can go back from the current transaction
+ * id till the age. And then, we check whether or not the xmin or
+ * catalog_xmin falls within the cutoff; if yes, return true, otherwise
+ * false.
+ */
+ cutoff = curr - replication_slot_xid_age;
+
+ if (!TransactionIdIsNormal(cutoff))
+ cutoff = FirstNormalTransactionId;
+
+ *xmin = InvalidTransactionId;
+ *catalog_xmin = InvalidTransactionId;
+
+ if (TransactionIdIsNormal(slot->data.xmin) &&
+ TransactionIdPrecedesOrEquals(slot->data.xmin, cutoff))
+ {
+ *xmin = slot->data.xmin;
+ return true;
+ }
+
+ if (TransactionIdIsNormal(slot->data.catalog_xmin) &&
+ TransactionIdPrecedesOrEquals(slot->data.catalog_xmin, cutoff))
+ {
+ *catalog_xmin = slot->data.catalog_xmin;
+ return true;
+ }
+
+ return false;
+}
+
/*
* Load all replication slots from disk into memory at server startup. This
* needs to be run before we start crash recovery.
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 861692c683..5d10dd1c8a 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3040,6 +3040,16 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_xid_age", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Age of the transaction ID at which a replication slot gets invalidated."),
+ gettext_noop("The transaction is the oldest transaction (including the one affecting the system catalogs) that a replication slot needs the database to retain.")
+ },
+ &replication_slot_xid_age,
+ 0, 0, 2000000000,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index deca3a4aeb..3fb6813195 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
+#replication_slot_xid_age = 0
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index dd56a77547..5ea73f9331 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -55,6 +55,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_WAL_LEVEL,
/* inactive slot timeout has occurred */
RS_INVAL_INACTIVE_TIMEOUT,
+ /* slot's xmin or catalog_xmin has reached the age */
+ RS_INVAL_XID_AGE,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -233,6 +235,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
extern PGDLLIMPORT int replication_slot_inactive_timeout;
+extern PGDLLIMPORT int replication_slot_xid_age;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
index 4663019c16..18300cfeca 100644
--- a/src/test/recovery/t/050_invalidate_slots.pl
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -89,7 +89,7 @@ $primary->reload;
# that nobody has acquired that slot yet, so due to
# replication_slot_inactive_timeout setting above it must get invalidated.
wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Set timeout on the standby also to check the synced slots don't get
# invalidated due to timeout on the standby.
@@ -129,7 +129,7 @@ $standby1->stop;
# Wait for the standby's replication slot to become inactive
wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Testcase end: Invalidate streaming standby's slot as well as logical failover
# slot on primary due to replication_slot_inactive_timeout. Also, check the
@@ -197,15 +197,280 @@ $subscriber->stop;
# Wait for the replication slot to become inactive and then invalidated due to
# timeout.
wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
- $inactive_timeout);
+ $inactive_timeout, 'inactive_timeout');
# Testcase end: Invalidate logical subscriber's slot due to
# replication_slot_inactive_timeout.
# =============================================================================
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot due to replication_slot_xid_age
+# GUC.
+
+# Prepare for the next test
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$primary->reload;
+
+# Create a standby linking to the primary using the replication slot
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+# Enable hs_feedback. The slot should gain an xmin. We set the status interval
+# so we'll see the results promptly.
+$standby2->append_conf(
+ 'postgresql.conf', q{
+primary_slot_name = 'sb2_slot'
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+});
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb2_slot', immediately_reserve := true);
+]);
+
+$standby2->start;
+
+# Create some content on primary to move xmin
+$primary->safe_psql('postgres',
+ "CREATE TABLE tab_int AS SELECT generate_series(1,10) AS a");
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby2);
+
+$primary->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NOT NULL AND catalog_xmin IS NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'sb2_slot';
+]) or die "Timed out waiting for slot sb2_slot xmin to advance";
+
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$primary->reload;
+
+# Stop standby to make the replication slot's xmin on primary to age
+$standby2->stop;
+
+$logstart = -s $primary->logfile;
+
+# Do some work to advance xids on primary
+advance_xids($primary, 'tab_int');
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($primary, 'sb2_slot', $logstart, 0, 'xid_aged');
+
+# Testcase end: Invalidate streaming standby's slot due to replication_slot_xid_age
+# GUC.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_xid_age GUC.
+
+$publisher = $primary;
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$publisher->reload;
+
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+));
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl2 (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl2 (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl2 VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+$publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub2 FOR TABLE test_tbl2");
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub2 CONNECTION '$publisher_connstr' PUBLICATION pub2 WITH (slot_name = 'lsub2_slot')"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub2');
+
+$result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl2");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NULL AND catalog_xmin IS NOT NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'lsub2_slot';
+]) or die "Timed out waiting for slot lsub2_slot catalog_xmin to advance";
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Do some work to advance xids on publisher
+advance_xids($publisher, 'test_tbl2');
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($publisher, 'lsub2_slot', $logstart, 0,
+ 'xid_aged');
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_xid_age GUC.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical slot on standby that's being synced from
+# the primary due to replication_slot_xid_age GUC.
+
+$publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 0;
+]);
+$publisher->reload;
+
+# Create a standby linking to the primary using the replication slot
+my $standby3 = PostgreSQL::Test::Cluster->new('standby3');
+$standby3->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+$standby3->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb3_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb3_slot', immediately_reserve := true);
+]);
+
+$standby3->start;
+
+my $standby3_logstart = -s $standby3->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby3);
+
+$subscriber->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+wal_receiver_status_interval = 1
+));
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl3 (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl3 (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl3 VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+$publisher->safe_psql('postgres',
+ "CREATE PUBLICATION pub3 FOR TABLE test_tbl3");
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub3 CONNECTION '$publisher_connstr' PUBLICATION pub3 WITH (slot_name = 'lsub3_sync_slot', failover = true)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub3');
+
+$result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl3");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->poll_query_until(
+ 'postgres', qq[
+ SELECT xmin IS NULL AND catalog_xmin IS NOT NULL
+ FROM pg_catalog.pg_replication_slots
+ WHERE slot_name = 'lsub3_sync_slot';
+])
+ or die "Timed out waiting for slot lsub3_sync_slot catalog_xmin to advance";
+
+# Synchronize the primary server slots to the standby
+$standby3->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced' and has got catalog_xmin from the primary.
+is( $standby3->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub3_sync_slot' AND synced AND NOT temporary AND
+ xmin IS NULL AND catalog_xmin IS NOT NULL;}
+ ),
+ "t",
+ 'logical slot has synced as true on standby');
+
+my $primary_catalog_xmin = $primary->safe_psql('postgres',
+ "SELECT catalog_xmin FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND catalog_xmin IS NOT NULL;"
+);
+
+my $stabdby3_catalog_xmin = $standby3->safe_psql('postgres',
+ "SELECT catalog_xmin FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND catalog_xmin IS NOT NULL;"
+);
+
+is($primary_catalog_xmin, $stabdby3_catalog_xmin,
+ "check catalog_xmin are same for primary slot and synced slot");
+
+# Enable XID age based invalidation on the standby. Note that we disabled the
+# same on the primary to check if the invalidation occurs for synced slot on
+# the standby.
+$standby3->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_xid_age = 500;
+]);
+$standby3->reload;
+
+$logstart = -s $standby3->logfile;
+
+# Do some work to advance xids on primary
+advance_xids($primary, 'test_tbl3');
+
+# Wait for standby to catch up with the above work
+$primary->wait_for_catchup($standby3);
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# XID age.
+wait_for_slot_invalidation($standby3, 'lsub3_sync_slot', $logstart, 0,
+ 'xid_aged');
+
+# Note that the replication slot on the primary is still active
+$result = $primary->safe_psql('postgres',
+ "SELECT COUNT(slot_name) = 1 FROM pg_replication_slots WHERE slot_name = 'lsub3_sync_slot' AND invalidation_reason IS NULL;"
+);
+
+is($result, 't', "check lsub3_sync_slot is still active on primary");
+
+# Testcase end: Invalidate logical slot on standby that's being synced from
+# the primary due to replication_slot_xid_age GUC.
+# =============================================================================
+
sub wait_for_slot_invalidation
{
- my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my ($node, $slot_name, $offset, $inactive_timeout, $reason) = @_;
my $name = $node->name;
# Wait for the replication slot to become inactive
@@ -231,14 +496,15 @@ sub wait_for_slot_invalidation
# for the slot to get invalidated.
sleep($inactive_timeout);
- check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset,
+ $reason);
# Wait for the inactive replication slot to be invalidated
$node->poll_query_until(
'postgres', qq[
SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
WHERE slot_name = '$slot_name' AND
- invalidation_reason = 'inactive_timeout';
+ invalidation_reason = '$reason';
])
or die
"Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
@@ -262,15 +528,33 @@ sub wait_for_slot_invalidation
# Check for invalidation of slot in server log
sub check_for_slot_invalidation_in_server_log
{
- my ($node, $slot_name, $offset) = @_;
+ my ($node, $slot_name, $offset, $reason) = @_;
my $name = $node->name;
my $invalidated = 0;
+ my $isrecovery =
+ $node->safe_psql('postgres', "SELECT pg_is_in_recovery()");
+
+ chomp($isrecovery);
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
- $node->safe_psql('postgres', "CHECKPOINT");
+ if ($reason eq 'xid_aged' && $isrecovery eq 'f')
+ {
+ $node->safe_psql('postgres', "VACUUM");
+ }
+ else
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ }
+
if ($node->log_contains(
"invalidating obsolete replication slot \"$slot_name\"",
+ $offset)
+ || $node->log_contains(
+ "The slot's xmin .* has reached the age .* specified by \"replication_slot_xid_age\".",
+ $offset)
+ || $node->log_contains(
+ "The slot's catalog_xmin .* has reached the age .* specified by \"replication_slot_xid_age\".",
$offset))
{
$invalidated = 1;
@@ -283,4 +567,25 @@ sub check_for_slot_invalidation_in_server_log
);
}
+# Do some work for advancing xids on a given node
+sub advance_xids
+{
+ my ($node, $table_name) = @_;
+
+ $node->safe_psql(
+ 'postgres', qq[
+ do \$\$
+ begin
+ for i in 10000..11000 loop
+ -- use an exception block so that each iteration eats an XID
+ begin
+ insert into $table_name values (i);
+ exception
+ when division_by_zero then null;
+ end;
+ end loop;
+ end\$\$;
+ ]);
+}
+
done_testing();
--
2.43.0
[application/octet-stream] v43-0001-Add-inactive_timeout-based-replication-slot-inva.patch (33.0K, ../../CALj2ACXe8+xSNdMXTMaSRWUwX7v61Ad4iddUwnn=djSwx3GLLg@mail.gmail.com/4-v43-0001-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From 8c035bcf94db49d09bbfb275018de3286e8522d7 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Thu, 29 Aug 2024 05:01:18 +0000
Subject: [PATCH v43 1/2] Add inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
max_slot_wal_keep_size is tricky. Because the amount of WAL a
customer generates, and their allocated storage will vary greatly
in production, making it difficult to pin down a one-size-fits-all
value. It is often easy for developers to set a timeout of say 1
or 2 or 3 days, after which the inactive slots get invalidated.
To achieve the above, postgres introduces a GUC allowing users
set inactive timeout. The replication slots that are inactive
for longer than specified amount of time get invalidated.
The invalidation check happens at various locations to help being
as latest as possible, these locations include the following:
- Whenever the slot is acquired and the slot acquisition errors
out if invalidated.
- During checkpoint
Note that this new invalidation mechanism won't kick-in for the
slots that are currently being synced from the primary to the
standby, because such synced slots are typically considered not
active (for them to be later considered as inactive) as they don't
perform logical decoding to produce the changes.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila
Reviewed-by: Ajin Cherian, Shveta Malik
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/202403260841.5jcv7ihniccy%40alvherre.pgsql
---
doc/src/sgml/config.sgml | 33 ++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 4 +-
src/backend/replication/slot.c | 183 ++++++++++-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 6 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 286 ++++++++++++++++++
13 files changed, 523 insertions(+), 20 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 12feac6087..113303a501 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4556,6 +4556,39 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during a checkpoint. The time since the slot has become
+ inactive is known from its
+ <structfield>inactive_since</structfield> value using which the
+ timeout is measured.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>).
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 634a4c0fab..9e00f7d184 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2618,6 +2618,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for the duration specified by
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 76de017635..758f0358b3 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -448,7 +448,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -667,7 +667,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index c290339af5..70093500fa 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -159,6 +161,13 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
+ ReplicationSlot *s,
+ XLogRecPtr oldestLSN,
+ Oid dboid,
+ TransactionId snapshotConflictHorizon,
+ bool *invalidated);
+
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
static void CreateSlotOnDisk(ReplicationSlot *slot);
@@ -535,12 +544,17 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if check_for_invalidation is true and the slot gets
+ * invalidated now or has been invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation)
{
ReplicationSlot *s;
int active_pid;
+ bool released_lock = false;
Assert(name != NULL);
@@ -615,6 +629,57 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ /*
+ * Check if the acquired slot needs to be invalidated. Error out if it
+ * gets invalidated now or has been invalidated previously, because
+ * there's no use in acquiring the invalidated slot.
+ *
+ * XXX: Currently we check for inactive_timeout invalidation here. We
+ * might need to check for other invalidations too.
+ */
+ if (check_for_invalidation)
+ {
+ bool invalidated = false;
+
+ released_lock = InvalidatePossiblyObsoleteSlot(RS_INVAL_INACTIVE_TIMEOUT,
+ s, 0, InvalidOid,
+ InvalidTransactionId,
+ &invalidated);
+
+ /*
+ * If the slot has been invalidated, recalculate the resource limits.
+ */
+ if (invalidated)
+ {
+ ReplicationSlotsComputeRequiredXmin(false);
+ ReplicationSlotsComputeRequiredLSN();
+ }
+
+ if (s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ /*
+ * Release the lock if it hasn't been already, to ensure smooth
+ * cleanup on error.
+ */
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive since %s for more than %d seconds specified by \"replication_slot_inactive_timeout\".",
+ timestamptz_to_str(s->inactive_since),
+ replication_slot_inactive_timeout)));
+ }
+ }
+
+ if (!released_lock)
+ LWLockRelease(ReplicationSlotControlLock);
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +850,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +877,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1501,7 +1566,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1531,6 +1597,13 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for more than %d seconds specified by \"replication_slot_inactive_timeout\"."),
+ timestamptz_to_str(inactive_since),
+ replication_slot_inactive_timeout);
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1574,6 +1647,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1581,6 +1655,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1591,6 +1666,18 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced)))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1644,6 +1731,39 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Quick exit if inactive timeout invalidation mechanism
+ * is disabled or slot is currently being used or the slot
+ * on standby is currently being synced from the primary.
+ *
+ * Note that we don't invalidate synced slots because,
+ * they are typically considered not active as they don't
+ * perform logical decoding to produce the changes.
+ */
+ if (replication_slot_inactive_timeout == 0 ||
+ s->inactive_since == 0 ||
+ (RecoveryInProgress() && s->data.synced))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1669,11 +1789,14 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so or if the slot is already ours,
+ * then mark it invalidated. Otherwise we'll signal the owning
+ * process, below, and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot != NULL &&
+ MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1728,7 +1851,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1774,7 +1898,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1797,6 +1922,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1849,7 +1975,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1907,6 +2034,38 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout inavalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index c7bfbb15e0..b1b7b075bd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -540,7 +540,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c5f1009f37..61a0e38715 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -844,7 +844,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1462,7 +1462,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index af227b1f24..861692c683 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a..deca3a4aeb 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index c2ee149fd6..dd56a77547 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -53,6 +53,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -230,6 +232,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -246,7 +249,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 712924c2fa..301be0f6c1 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -10,6 +10,7 @@ tests += {
'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
},
'tests': [
+ 't/050_invalidate_slots.pl',
't/001_stream_rep.pl',
't/002_archiving.pl',
't/003_recovery_targets.pl',
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..4663019c16
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,286 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start: Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to replication_slot_inactive_timeout. Also,
+# check the logical failover slot synced on to the standby doesn't invalidate
+# the slot on its own, but gets the invalidated state from the remote slot on
+# the primary.
+
+# Initialize primary node
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid checkpoint during the test, otherwise, the test can get unpredictable
+$primary->append_conf(
+ 'postgresql.conf', q{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create a standby linking to the primary using the replication slot
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr_1 = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb1_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+my $standby1_logstart = -s $standby1->logfile;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Synchronize the primary server slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced'.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot lsub1_sync_slot has synced as true on standby');
+
+my $logstart = -s $primary->logfile;
+my $inactive_timeout = 2;
+
+# Set timeout so that the next checkpoint will invalidate the inactive
+# replication slot.
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for the logical failover slot to become inactive on the primary. Note
+# that nobody has acquired that slot yet, so due to
+# replication_slot_inactive_timeout setting above it must get invalidated.
+wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
+ $inactive_timeout);
+
+# Set timeout on the standby also to check the synced slots don't get
+# invalidated due to timeout on the standby.
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$standby1->reload;
+
+# Now, sync the logical failover slot from the remote slot on the primary.
+# Note that the remote slot has already been invalidated due to inactive
+# timeout. Now, the standby must also see it as invalidated.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for replication slot lsub1_sync_slot invalidation to be synced on standby";
+
+# Synced slot mustn't get invalidated on the standby even after a checkpoint,
+# it must sync invalidation from the primary. So, we must not see the slot's
+# invalidation message in server log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned slot lsub1_sync_slot has not been invalidated on the standby'
+);
+
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate streaming standby's slot as well as logical failover
+# slot on primary due to replication_slot_inactive_timeout. Also, check the
+# logical failover slot synced on to the standby doesn't invalidate the slot on
+# its own, but gets the invalidated state from the remote slot on the primary.
+# =============================================================================
+
+# =============================================================================
+# Testcase start: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+
+my $publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber node
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
+
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for slot $slot_name to become inactive on node $name";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of slot $slot_name to be updated on node $name";
+
+ # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
+ # for the slot to get invalidated.
+ sleep($inactive_timeout);
+
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
+
+ # Check that the invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot_name', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot_name"/,
+ "detected error upon trying to acquire invalidated slot $slot_name on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot_name on node $name";
+}
+
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $name = $node->name;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot_name\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot_name invalidation has been logged on node $name"
+ );
+}
+
+done_testing();
--
2.43.0
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-08-30 02:43 ` Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: Peter Smith @ 2024-08-30 02:43 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi, here are some review comments for patch v43-0001.
======
Commit message
1.
... introduces a GUC allowing users set inactive timeout.
~
1a. You should give the name of the new GUC in the commit message.
1b. /set/to set/
======
doc/src/sgml/config.sgml
GUC "replication_slot_inactive_timeout"
2.
Invalidates replication slots that are inactive for longer than
specified amount of time
nit - suggest use similar wording as the prior GUC (wal_sender_timeout):
Invalidate replication slots that are inactive for longer than this
amount of time.
~
3.
This invalidation check happens either when the slot is acquired for
use or during a checkpoint. The time since the slot has become
inactive is known from its inactive_since value using which the
timeout is measured.
nit - the wording is too complicated. suggest:
The timeout check occurs when the slot is next acquired for use, or
during a checkpoint. The slot's 'inactive_since' field value is when
the slot became inactive.
~
4.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby that are being synced from a
primary server (whose synced field is true).
nit - that word "whose" seems ambiguous. suggest:
(e.g. the standby slot has 'synced' field true).
======
doc/src/sgml/system-views.sgml
5.
inactive_timeout means that the slot has been inactive for the
duration specified by replication_slot_inactive_timeout parameter.
nit - suggestion ("longer than"):
... the slot has been inactive for longer than the duration specified
by the replication_slot_inactive_timeout parameter.
======
src/backend/replication/slot.c
6.
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
IMO this #define belongs in the slot.h, immediately below where the
enum is defined.
~~~
7. ReplicationSlotAcquire:
I had a fundamental question about this logic.
IIUC the purpose of the patch was to invalidate replication slots that
have been inactive for too long.
So, it makes sense to me that some periodic processing (e.g.
CheckPointReplicationSlots) might do a sweep over all the slots, and
invalidate the too-long-inactive ones that it finds.
OTOH, it seemed quite strange to me that the patch logic is also
detecting and invalidating inactive slots during the
ReplicationSlotAcquire function. This is kind of saying "ERROR -
sorry, because this was inactive for too long you can't have it" at
the very moment that you wanted to use it again! IIUC such a slot
would be invalidated by the function InvalidatePossiblyObsoleteSlot(),
but therein lies my doubt -- how can the slot be considered as
"obsolete" when we are in the very act of trying to acquire/use it?
I guess it might be argued this is not so different to the scenario of
attempting to acquire a slot that had been invalidated momentarily
before during checkpoint processing. But, somehow that scenario seems
more like bad luck to me, versus ReplicationSlotAcquire() deliberately
invalidating something we *know* is wanted.
~
8.
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive
since %s for more than %d seconds specified by
\"replication_slot_inactive_timeout\".",
+ timestamptz_to_str(s->inactive_since),
+ replication_slot_inactive_timeout)));
nit - IMO the info should be split into errdetail + errhint. Like this:
errdetail("The slot became invalid because it was inactive since %s,
which is more than %d seconds ago."...)
errhint("You might need to increase \"%s\".",
"replication_slot_inactive_timeout")
~~~
9. ReportSlotInvalidation
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for more than %d seconds
specified by \"replication_slot_inactive_timeout\"."),
+ timestamptz_to_str(inactive_since),
+ replication_slot_inactive_timeout);
+ break;
IMO this error in ReportSlotInvalidation() should be the same as the
other one from ReplicationSlotAcquire(), which I suggested above
(comment #8) should include a hint. Also, including a hint here will
make this new message consistent with the other errhint (for
"max_slot_wal_keep_size") that is already in this function.
~~~
10. InvalidatePossiblyObsoleteSlot
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced)))
10a. Everything here is && so this has some redundant parentheses.
10b. Actually, IMO this complicated condition is overkill. Won't it be
better to just unconditionally assign
now = GetCurrentTimestamp(); here?
~
11.
+ * Note that we don't invalidate synced slots because,
+ * they are typically considered not active as they don't
+ * perform logical decoding to produce the changes.
nit - tweaked punctuation
~
12.
+ * If the slot can be acquired, do so or if the slot is already ours,
+ * then mark it invalidated. Otherwise we'll signal the owning
+ * process, below, and retry.
nit - tidied this comment. Suggestion:
If the slot can be acquired, do so and mark it as invalidated. If the
slot is already ours, mark it as invalidated. Otherwise, we'll signal
the owning process below and retry.
~
13.
+ if (active_pid == 0 ||
+ (MyReplicationSlot != NULL &&
+ MyReplicationSlot == s &&
+ active_pid == MyProcPid))
You are already checking MyReplicationSlot == s here, so that extra
check for MyReplicationSlot != NULL is redundant, isn't it?
~~~
14. CheckPointReplicationSlots
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
nit - /Also, invalidate slots/Also, invalidate obsolete slots/
======
src/backend/utils/misc/guc_tables.c
15.
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time to wait before invalidating an "
+ "inactive replication slot."),
nit - that is maybe a bit misleading because IIUC there is no real
"waiting" happening anywhere. Suggest:
Sets the amount of time a replication slot can remain inactive before
it will be invalidated.
======
Please take a look at the attached top-up patches. These include
changes for many of the nits above.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 7009350..c96ae53 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -671,9 +671,10 @@ retry:
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("can no longer get changes from replication slot \"%s\"",
NameStr(s->data.name)),
- errdetail("This slot has been invalidated because it was inactive since %s for more than %d seconds specified by \"replication_slot_inactive_timeout\".",
+ errdetail("The slot became invalid because it was inactive since %s, which is more than %d seconds ago.",
timestamptz_to_str(s->inactive_since),
- replication_slot_inactive_timeout)));
+ replication_slot_inactive_timeout),
+ errhint("You might need to increase \"%s\".", "replication_slot_inactive_timeout")));
}
}
@@ -1738,9 +1739,9 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* is disabled or slot is currently being used or the slot
* on standby is currently being synced from the primary.
*
- * Note that we don't invalidate synced slots because,
- * they are typically considered not active as they don't
- * perform logical decoding to produce the changes.
+ * Note that we don't invalidate synced slots because
+ * they are typically considered not active, as they don't
+ * perform logical decoding to produce changes.
*/
if (replication_slot_inactive_timeout == 0 ||
s->inactive_since == 0 ||
@@ -1789,9 +1790,9 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so or if the slot is already ours,
- * then mark it invalidated. Otherwise we'll signal the owning
- * process, below, and retry.
+ * If the slot can be acquired, do so and mark it as invalidated.
+ * If the slot is already ours, mark it as invalidated.
+ * Otherwise, we'll signal the owning process below and retry.
*/
if (active_pid == 0 ||
(MyReplicationSlot != NULL &&
@@ -1975,7 +1976,7 @@ restart:
}
/*
- * Flush all replication slots to disk. Also, invalidate slots during
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
* non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 861692c..73a5824 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3030,8 +3030,9 @@ struct config_int ConfigureNamesInt[] =
{
{"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
- gettext_noop("Sets the amount of time to wait before invalidating an "
- "inactive replication slot."),
+
+ gettext_noop("Sets the amount of time a replication slot can remain "
+ "inactive before it will be invalidated."),
NULL,
GUC_UNIT_S
},
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index fbbacbe..bd3ce5a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4568,8 +4568,8 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</term>
<listitem>
<para>
- Invalidates replication slots that are inactive for longer than
- specified amount of time. If this value is specified without units,
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
it is taken as seconds. A value of zero (which is default) disables
the timeout mechanism. This parameter can only be set in
the <filename>postgresql.conf</filename> file or on the server
@@ -4577,18 +4577,16 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</para>
<para>
- This invalidation check happens either when the slot is acquired
- for use or during a checkpoint. The time since the slot has become
- inactive is known from its
- <structfield>inactive_since</structfield> value using which the
- timeout is measured.
+ The timeout check occurs when the slot is next acquired for use, or
+ during a checkpoint. The slot's <structfield>inactive_since</structfield>
+ field value is when the slot became inactive.
</para>
<para>
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby that are being synced from a
- primary server (whose <structfield>synced</structfield> field is
- <literal>true</literal>).
+ primary server (e.g. the standby slot <structfield>synced</structfield>
+ field is true).
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 9e00f7d..f230e6e 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2621,7 +2621,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<listitem>
<para>
<literal>inactive_timeout</literal> means that the slot has been
- inactive for the duration specified by
+ inactive for longer than the duration specified by
<xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
</para>
</listitem>
Attachments:
[text/plain] PS_NITPICKS_20240830_CODE_V430001.txt (3.0K, ../../CAHut+PuFzCHPCiZbpoQX59kgZbebuWT0gR0O7rOe4t_sdYu=OA@mail.gmail.com/2-PS_NITPICKS_20240830_CODE_V430001.txt)
download | inline diff:
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 7009350..c96ae53 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -671,9 +671,10 @@ retry:
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("can no longer get changes from replication slot \"%s\"",
NameStr(s->data.name)),
- errdetail("This slot has been invalidated because it was inactive since %s for more than %d seconds specified by \"replication_slot_inactive_timeout\".",
+ errdetail("The slot became invalid because it was inactive since %s, which is more than %d seconds ago.",
timestamptz_to_str(s->inactive_since),
- replication_slot_inactive_timeout)));
+ replication_slot_inactive_timeout),
+ errhint("You might need to increase \"%s\".", "replication_slot_inactive_timeout")));
}
}
@@ -1738,9 +1739,9 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* is disabled or slot is currently being used or the slot
* on standby is currently being synced from the primary.
*
- * Note that we don't invalidate synced slots because,
- * they are typically considered not active as they don't
- * perform logical decoding to produce the changes.
+ * Note that we don't invalidate synced slots because
+ * they are typically considered not active, as they don't
+ * perform logical decoding to produce changes.
*/
if (replication_slot_inactive_timeout == 0 ||
s->inactive_since == 0 ||
@@ -1789,9 +1790,9 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so or if the slot is already ours,
- * then mark it invalidated. Otherwise we'll signal the owning
- * process, below, and retry.
+ * If the slot can be acquired, do so and mark it as invalidated.
+ * If the slot is already ours, mark it as invalidated.
+ * Otherwise, we'll signal the owning process below and retry.
*/
if (active_pid == 0 ||
(MyReplicationSlot != NULL &&
@@ -1975,7 +1976,7 @@ restart:
}
/*
- * Flush all replication slots to disk. Also, invalidate slots during
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
* non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 861692c..73a5824 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3030,8 +3030,9 @@ struct config_int ConfigureNamesInt[] =
{
{"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
- gettext_noop("Sets the amount of time to wait before invalidating an "
- "inactive replication slot."),
+
+ gettext_noop("Sets the amount of time a replication slot can remain "
+ "inactive before it will be invalidated."),
NULL,
GUC_UNIT_S
},
[text/plain] PS_NITPICKS_20240829_DOCS_v430001.txt (2.4K, ../../CAHut+PuFzCHPCiZbpoQX59kgZbebuWT0gR0O7rOe4t_sdYu=OA@mail.gmail.com/3-PS_NITPICKS_20240829_DOCS_v430001.txt)
download | inline diff:
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index fbbacbe..bd3ce5a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4568,8 +4568,8 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</term>
<listitem>
<para>
- Invalidates replication slots that are inactive for longer than
- specified amount of time. If this value is specified without units,
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
it is taken as seconds. A value of zero (which is default) disables
the timeout mechanism. This parameter can only be set in
the <filename>postgresql.conf</filename> file or on the server
@@ -4577,18 +4577,16 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</para>
<para>
- This invalidation check happens either when the slot is acquired
- for use or during a checkpoint. The time since the slot has become
- inactive is known from its
- <structfield>inactive_since</structfield> value using which the
- timeout is measured.
+ The timeout check occurs when the slot is next acquired for use, or
+ during a checkpoint. The slot's <structfield>inactive_since</structfield>
+ field value is when the slot became inactive.
</para>
<para>
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby that are being synced from a
- primary server (whose <structfield>synced</structfield> field is
- <literal>true</literal>).
+ primary server (e.g. the standby slot <structfield>synced</structfield>
+ field is true).
</para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 9e00f7d..f230e6e 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2621,7 +2621,7 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<listitem>
<para>
<literal>inactive_timeout</literal> means that the slot has been
- inactive for the duration specified by
+ inactive for longer than the duration specified by
<xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
</para>
</listitem>
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
@ 2024-08-31 08:15 ` Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-02 09:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-09-03 06:55 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
0 siblings, 4 replies; 98+ messages in thread
From: Bharath Rupireddy @ 2024-08-31 08:15 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Thanks for looking into this.
On Fri, Aug 30, 2024 at 8:13 AM Peter Smith <[email protected]> wrote:
>
> ======
> Commit message
>
> 1.
> ... introduces a GUC allowing users set inactive timeout.
>
> ~
>
> 1a. You should give the name of the new GUC in the commit message.
Modified.
> 1b. /set/to set/
Reworded the commit message.
> ======
> doc/src/sgml/config.sgml
>
> GUC "replication_slot_inactive_timeout"
>
> 2.
> Invalidates replication slots that are inactive for longer than
> specified amount of time
>
> nit - suggest use similar wording as the prior GUC (wal_sender_timeout):
> Invalidate replication slots that are inactive for longer than this
> amount of time.
Modified.
> 3.
> This invalidation check happens either when the slot is acquired for
> use or during a checkpoint. The time since the slot has become
> inactive is known from its inactive_since value using which the
> timeout is measured.
>
> nit - the wording is too complicated. suggest:
> The timeout check occurs when the slot is next acquired for use, or
> during a checkpoint. The slot's 'inactive_since' field value is when
> the slot became inactive.
> 4.
> Note that the inactive timeout invalidation mechanism is not
> applicable for slots on the standby that are being synced from a
> primary server (whose synced field is true).
>
> nit - that word "whose" seems ambiguous. suggest:
> (e.g. the standby slot has 'synced' field true).
Reworded.
> ======
> doc/src/sgml/system-views.sgml
>
> 5.
> inactive_timeout means that the slot has been inactive for the
> duration specified by replication_slot_inactive_timeout parameter.
>
> nit - suggestion ("longer than"):
> ... the slot has been inactive for longer than the duration specified
> by the replication_slot_inactive_timeout parameter.
Modified.
> ======
> src/backend/replication/slot.c
>
> 6.
> /* Maximum number of invalidation causes */
> -#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
> +#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
>
> IMO this #define belongs in the slot.h, immediately below where the
> enum is defined.
Please check the commit that introduced it -
https://www.postgresql.org/message-id/ZdU3CHqza9XJw4P-%40paquier.xyz.
It is kept in the file where it's used.
> 7. ReplicationSlotAcquire:
>
> I had a fundamental question about this logic.
>
> IIUC the purpose of the patch was to invalidate replication slots that
> have been inactive for too long.
>
> So, it makes sense to me that some periodic processing (e.g.
> CheckPointReplicationSlots) might do a sweep over all the slots, and
> invalidate the too-long-inactive ones that it finds.
>
> OTOH, it seemed quite strange to me that the patch logic is also
> detecting and invalidating inactive slots during the
> ReplicationSlotAcquire function. This is kind of saying "ERROR -
> sorry, because this was inactive for too long you can't have it" at
> the very moment that you wanted to use it again! IIUC such a slot
> would be invalidated by the function InvalidatePossiblyObsoleteSlot(),
> but therein lies my doubt -- how can the slot be considered as
> "obsolete" when we are in the very act of trying to acquire/use it?
>
> I guess it might be argued this is not so different to the scenario of
> attempting to acquire a slot that had been invalidated momentarily
> before during checkpoint processing. But, somehow that scenario seems
> more like bad luck to me, versus ReplicationSlotAcquire() deliberately
> invalidating something we *know* is wanted.
Hm. TBH, there's no real reason for invalidating the slot in
ReplicationSlotAcquire(). My thinking back then was to take this
opportunity to do some work. I agree to leave the invalidation work to
the checkpointer. However, I still think ReplicationSlotAcquire()
should error out if the slot has already been invalidated similar to
"can no longer get changes from replication slot \"%s\" for
wal_removed.
> 8.
> + ereport(ERROR,
> + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
> + errmsg("can no longer get changes from replication slot \"%s\"",
> + NameStr(s->data.name)),
> + errdetail("This slot has been invalidated because it was inactive
> since %s for more than %d seconds specified by
> \"replication_slot_inactive_timeout\".",
> + timestamptz_to_str(s->inactive_since),
> + replication_slot_inactive_timeout)));
>
> nit - IMO the info should be split into errdetail + errhint. Like this:
> errdetail("The slot became invalid because it was inactive since %s,
> which is more than %d seconds ago."...)
> errhint("You might need to increase \"%s\".",
> "replication_slot_inactive_timeout")
"invalid" is being covered by errmsg "invalidating obsolete
replication slot", so no need to duplicate it in errdetail.
> 9. ReportSlotInvalidation
>
> + appendStringInfo(&err_detail,
> + _("The slot has been inactive since %s for more than %d seconds
> specified by \"replication_slot_inactive_timeout\"."),
> + timestamptz_to_str(inactive_since),
> + replication_slot_inactive_timeout);
> + break;
>
> IMO this error in ReportSlotInvalidation() should be the same as the
> other one from ReplicationSlotAcquire(), which I suggested above
> (comment #8) should include a hint. Also, including a hint here will
> make this new message consistent with the other errhint (for
> "max_slot_wal_keep_size") that is already in this function.
Not exactly the same but similar. Because ReportSlotInvalidation()
errmsg has an "invalidating" component, whereas errmsg in
ReplicationSlotAcquire doesn't. Please check latest wordings.
> 10. InvalidatePossiblyObsoleteSlot
>
> + if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
> + (replication_slot_inactive_timeout > 0 &&
> + s->inactive_since > 0 &&
> + !(RecoveryInProgress() && s->data.synced)))
>
> 10a. Everything here is && so this has some redundant parentheses.
Removed.
> 10b. Actually, IMO this complicated condition is overkill. Won't it be
> better to just unconditionally assign
> now = GetCurrentTimestamp(); here?
GetCurrentTimestamp() can get costlier on certain platforms. I think
the fields checking in the condition are pretty straight forward -
e.g. !RecoveryInProgress() server not in recovery, !s->data.synced
slot is not being synced and so on. Added a macro
IsInactiveTimeoutSlotInvalidationApplicable() for better readability
in two places.
> 11.
> + * Note that we don't invalidate synced slots because,
> + * they are typically considered not active as they don't
> + * perform logical decoding to produce the changes.
>
> nit - tweaked punctuation
Used the consistent wording in the commit message, docs and code comments.
> 12.
> + * If the slot can be acquired, do so or if the slot is already ours,
> + * then mark it invalidated. Otherwise we'll signal the owning
> + * process, below, and retry.
>
> nit - tidied this comment. Suggestion:
> If the slot can be acquired, do so and mark it as invalidated. If the
> slot is already ours, mark it as invalidated. Otherwise, we'll signal
> the owning process below and retry.
Modified.
> 13.
> + if (active_pid == 0 ||
> + (MyReplicationSlot != NULL &&
> + MyReplicationSlot == s &&
> + active_pid == MyProcPid))
>
> You are already checking MyReplicationSlot == s here, so that extra
> check for MyReplicationSlot != NULL is redundant, isn't it?
Removed.
> 14. CheckPointReplicationSlots
>
> /*
> - * Flush all replication slots to disk.
> + * Flush all replication slots to disk. Also, invalidate slots during
> + * non-shutdown checkpoint.
> *
> * It is convenient to flush dirty replication slots at the time of checkpoint.
> * Additionally, in case of a shutdown checkpoint, we also identify the slots
>
> nit - /Also, invalidate slots/Also, invalidate obsolete slots/
Modified.
> 15.
> + {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
> + gettext_noop("Sets the amount of time to wait before invalidating an "
> + "inactive replication slot."),
>
> nit - that is maybe a bit misleading because IIUC there is no real
> "waiting" happening anywhere. Suggest:
> Sets the amount of time a replication slot can remain inactive before
> it will be invalidated.
Modified.
Please find the attached v44 patch with the above changes. I will
include the 0002 xid_age based invalidation patch later.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/x-patch] v44-0001-Add-inactive_timeout-based-replication-slot-inva.patch (33.5K, ../../CALj2ACUJg2faYd-_Wr3NWoj_GtbW5Bje3G9+o595GyEhcBkbrg@mail.gmail.com/2-v44-0001-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From d0ee643d29f7df6fa39581b4c9304f327c79256a Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Sat, 31 Aug 2024 07:43:22 +0000
Subject: [PATCH v44] Add inactive_timeout based replication slot invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage for instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named replication_slot_inactive_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are inactive for longer than this amount of
time.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from primary server (i.e., standby slots having 'synced' field
true). Because such synced slots are typically considered not
active (for them to be later considered as inactive) as they don't
perform logical decoding to produce the changes.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila
Reviewed-by: Ajin Cherian, Shveta Malik, Peter Smith
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/202403260841.5jcv7ihniccy%40alvherre.pgsql
---
doc/src/sgml/config.sgml | 36 +++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 4 +-
src/backend/replication/slot.c | 171 ++++++++++-
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 6 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 283 ++++++++++++++++++
13 files changed, 508 insertions(+), 23 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 0aec11f443..970b496e39 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4556,6 +4556,42 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during checkpoint. The time since the slot has become
+ inactive is known from its
+ <structfield>inactive_since</structfield> value using which the
+ timeout is measured.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from primary server (i.e., standby slots having
+ <structfield>synced</structfield> field <literal>true</literal>).
+ Because such synced slots are typically considered not active
+ (for them to be later considered as inactive) as they don't perform
+ logical decoding to produce the changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 634a4c0fab..f230e6e572 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2618,6 +2618,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for longer than the duration specified by
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 51072297fd..25c6a68b54 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -448,7 +448,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -667,7 +667,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 0a03776156..26448ecbfd 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -131,6 +132,12 @@ StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
#define SLOT_MAGIC 0x1051CA1 /* format identifier */
#define SLOT_VERSION 5 /* version for new files */
+#define IsInactiveTimeoutSlotInvalidationApplicable(s) \
+ (replication_slot_inactive_timeout > 0 && \
+ s->inactive_since > 0 && \
+ !RecoveryInProgress() && \
+ !s->data.synced)
+
/* Control array for replication slot management */
ReplicationSlotCtlData *ReplicationSlotCtl = NULL;
@@ -140,6 +147,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -159,6 +167,13 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
+ ReplicationSlot *s,
+ XLogRecPtr oldestLSN,
+ Oid dboid,
+ TransactionId snapshotConflictHorizon,
+ bool *invalidated);
+
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
static void CreateSlotOnDisk(ReplicationSlot *slot);
@@ -535,9 +550,13 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if check_for_invalidation is true and the slot has been
+ * invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +634,25 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * Error out if the slot has been invalidated previously. Because there's
+ * no use in acquiring the invalidated slot.
+ */
+ if (check_for_invalidation &&
+ s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("The slot became invalid because it was inactive since %s, which is more than %d seconds ago.",
+ timestamptz_to_str(s->inactive_since),
+ replication_slot_inactive_timeout),
+ errhint("You might need to increase \"%s\".",
+ "replication_slot_inactive_timeout.")));
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +823,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +850,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1501,10 +1539,11 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
- bool hint = false;
+ StringInfo err_hint = NULL;
initStringInfo(&err_detail);
@@ -1514,13 +1553,16 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
{
unsigned long long ex = oldestLSN - restart_lsn;
- hint = true;
appendStringInfo(&err_detail,
ngettext("The slot's restart_lsn %X/%X exceeds the limit by %llu byte.",
"The slot's restart_lsn %X/%X exceeds the limit by %llu bytes.",
ex),
LSN_FORMAT_ARGS(restart_lsn),
ex);
+
+ err_hint = makeStringInfo();
+ appendStringInfo(err_hint,
+ _("You might need to increase \"%s\"."), "max_slot_wal_keep_size");
break;
}
case RS_INVAL_HORIZON:
@@ -1531,6 +1573,17 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot became inactive since %s, which is more than %d seconds ago."),
+ timestamptz_to_str(inactive_since),
+ replication_slot_inactive_timeout);
+
+ err_hint = makeStringInfo();
+ appendStringInfo(err_hint,
+ _("You might need to increase \"%s\"."), "replication_slot_inactive_timeout");
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1542,9 +1595,12 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
errmsg("invalidating obsolete replication slot \"%s\"",
NameStr(slotname)),
errdetail_internal("%s", err_detail.data),
- hint ? errhint("You might need to increase \"%s\".", "max_slot_wal_keep_size") : 0);
+ (err_hint != NULL) ? errhint("%s", err_hint->data) : 0);
pfree(err_detail.data);
+
+ if (err_hint != NULL)
+ destroyStringInfo(err_hint);
}
/*
@@ -1574,6 +1630,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1581,6 +1638,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1591,6 +1649,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ IsInactiveTimeoutSlotInvalidationApplicable(s))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1644,6 +1712,41 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Quick exit if inactive timeout invalidation mechanism
+ * is disabled or slot is currently being used or the
+ * server is in recovery mode or the slot on standby is
+ * currently being synced from the primary.
+ *
+ * Note that the inactive timeout invalidation mechanism
+ * is not applicable for slots on the standby server that
+ * are being synced from primary server. Because such
+ * synced slots are typically considered not active (for
+ * them to be later considered as inactive) as they don't
+ * perform logical decoding to produce the changes.
+ */
+ if (!IsInactiveTimeoutSlotInvalidationApplicable(s))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1669,11 +1772,13 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1728,7 +1833,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1774,7 +1880,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1797,6 +1904,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1849,7 +1957,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1907,6 +2016,38 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index c7bfbb15e0..b1b7b075bd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -540,7 +540,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c5f1009f37..61a0e38715 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -844,7 +844,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1462,7 +1462,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 521ec5591c..675eb115ac 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a..deca3a4aeb 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 45582cf9d8..431cc08c99 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -233,6 +235,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -249,7 +252,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 712924c2fa..301be0f6c1 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -10,6 +10,7 @@ tests += {
'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
},
'tests': [
+ 't/050_invalidate_slots.pl',
't/001_stream_rep.pl',
't/002_archiving.pl',
't/003_recovery_targets.pl',
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..fa6a12a12d
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,283 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start
+#
+# Invalidate streaming standby's slot as well as logical
+# failover slot on primary due to replication_slot_inactive_timeout. Also,
+# check the logical failover slot synced on to the standby doesn't invalidate
+# the slot on its own, but gets the invalidated state from the remote slot on
+# the primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', q{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr_1 = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb1_slot'
+primary_conninfo = '$connstr_1 dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('lsub1_sync_slot', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb1_slot', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Synchronize the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby and is
+# flagged as 'synced'.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot lsub1_sync_slot has synced as true on standby');
+
+my $standby1_logstart = -s $standby1->logfile;
+my $logstart = -s $primary->logfile;
+my $inactive_timeout = 2;
+
+# Set timeout so that the next checkpoint will invalidate the inactive
+# replication slot.
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for the logical failover slot to become inactive on the primary. Note
+# that nobody has acquired that slot yet, so due to
+# replication_slot_inactive_timeout setting above it must get invalidated.
+wait_for_slot_invalidation($primary, 'lsub1_sync_slot', $logstart,
+ $inactive_timeout);
+
+# Set timeout on the standby also to check the synced slots don't get
+# invalidated due to timeout on the standby.
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '2s';
+]);
+$standby1->reload;
+
+# Now, sync the logical failover slot from the remote slot on the primary.
+# Note that the remote slot has already been invalidated due to inactive
+# timeout. Now, the standby must also see it as invalidated.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for lsub1_sync_slot invalidation to be synced on standby";
+
+# Synced slot mustn't get invalidated on the standby even after a checkpoint,
+# it must sync invalidation from the primary. So, we must not see the slot's
+# invalidation message in server log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned lsub1_sync_slot has not been invalidated on the standby'
+);
+
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# =============================================================================
+# Testcase start
+# Invalidate logical subscriber's slot due to replication_slot_inactive_timeout.
+
+my $publisher = $primary;
+
+# Prepare for the next test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
+
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot_name, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for slot $slot_name to become inactive on node $name";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of slot $slot_name to be updated on node $name";
+
+ # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
+ # for the slot to get invalidated.
+ sleep($inactive_timeout);
+
+ check_for_slot_invalidation_in_server_log($node, $slot_name, $offset);
+
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive slot $slot_name to be invalidated on node $name";
+
+ # Check that the invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot_name', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot_name"/,
+ "detected error upon trying to acquire invalidated slot $slot_name on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot_name on node $name";
+}
+
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{
+ my ($node, $slot_name, $offset) = @_;
+ my $name = $node->name;
+ my $invalidated = 0;
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot_name\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot_name invalidation has been logged on node $name"
+ );
+}
+
+done_testing();
--
2.43.0
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-02 08:06 ` Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
3 siblings, 1 reply; 98+ messages in thread
From: Peter Smith @ 2024-09-02 08:06 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi. Thanks for addressing my previous review comments.
Here are some review comments for v44-0001.
======
Commit message.
1.
Because such synced slots are typically considered not
active (for them to be later considered as inactive) as they don't
perform logical decoding to produce the changes.
~
This sentence is bad grammar. The docs have the same wording, so
please see my doc review comment #4 suggestion below.
======
doc/src/sgml/config.sgml
2.
+ <para>
+ Invalidates replication slots that are inactive for longer than
+ specified amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the timeout mechanism. This parameter can only be set in
+ the <filename>postgresql.conf</filename> file or on the server
+ command line.
+ </para>
+
nit - This is OK as-is, but OTOH why not make the wording consistent
with the previous GUC description? (e.g. see my v43 [1] #2 review
comment)
~~~
3.
+ <para>
+ This invalidation check happens either when the slot is acquired
+ for use or during checkpoint. The time since the slot has become
+ inactive is known from its
+ <structfield>inactive_since</structfield> value using which the
+ timeout is measured.
+ </para>
+
I felt this is slightly misleading because slot acquiring has nothing
to do with setting the slot invalidation anymore. Furthermore, the 2nd
sentence is bad grammar.
nit - IMO something simple like the following rewording can address
both of those points:
Slot invalidation due to inactivity timeout occurs during checkpoint.
The duration of slot inactivity is calculated using the slot's
<structfield>inactive_since</structfield> field value.
~
4.
+ Because such synced slots are typically considered not active
+ (for them to be later considered as inactive) as they don't perform
+ logical decoding to produce the changes.
That sentence has bad grammar.
nit – suggest a much simpler replacement:
Synced slots are always considered to be inactive because they don't
perform logical decoding to produce changes.
======
src/backend/replication/slot.c
5.
+#define IsInactiveTimeoutSlotInvalidationApplicable(s) \
+ (replication_slot_inactive_timeout > 0 && \
+ s->inactive_since > 0 && \
+ !RecoveryInProgress() && \
+ !s->data.synced)
+
5a.
I felt this would be better implemented as an inline function. Then it
can be commented on properly to explain the parts of the condition.
e.g. the large comment currently in InvalidatePossiblyObsoleteSlot()
would be more appropriate in this function.
~
5b.
The name is very long. Can't it be something shorter/simpler like:
'IsSlotATimeoutCandidate()'
~~~
6. ReplicationSlotAcquire
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation)
nit - Previously this new parameter really did mean to "check" for
[and set the slot] invalidation. But now I suggest renaming it to
'error_if_invalid' to properly reflect the new usage. And also in the
slot.h.
~
7.
+ /*
+ * Error out if the slot has been invalidated previously. Because there's
+ * no use in acquiring the invalidated slot.
+ */
nit - The comment is contrary to the code. If there was no reason to
skip this error, then you would not have the new parameter allowing
you to skip this error. I suggest just repeating the same comment as
in the function header.
~~~
8. ReportSlotInvalidation
nit - Added some blank lines for consistency.
~~~
9. InvalidatePossiblyObsoleteSlot
+ /*
+ * Quick exit if inactive timeout invalidation mechanism
+ * is disabled or slot is currently being used or the
+ * server is in recovery mode or the slot on standby is
+ * currently being synced from the primary.
+ *
+ * Note that the inactive timeout invalidation mechanism
+ * is not applicable for slots on the standby server that
+ * are being synced from primary server. Because such
+ * synced slots are typically considered not active (for
+ * them to be later considered as inactive) as they don't
+ * perform logical decoding to produce the changes.
+ */
+ if (!IsInactiveTimeoutSlotInvalidationApplicable(s))
+ break;
9a.
Consistency is good (commit message, docs and code comments for this),
but the added sentence has bad grammar. Please see the docs review
comment #4 above for some alternate phrasing.
~
9b.
Now that this logic is moved into a macro (I suggested it should be an
inline function) IMO this comment does not belong here anymore because
it is commenting code that you cannot see. Instead, this comment (or
something like it) should be as comments within the new function.
======
src/include/replication/slot.h
10.
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool check_for_invalidation);
Change the new param name as described in the earlier review comment.
======
src/test/recovery/t/050_invalidate_slots.pl
~~~
Please refer to the attached file which implements some of the nits
mentioned above.
======
[1] v43 review -
https://www.postgresql.org/message-id/CAHut%2BPuFzCHPCiZbpoQX59kgZbebuWT0gR0O7rOe4t_sdYu%3DOA%40mail...
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 970b496..0537714 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4564,8 +4564,8 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</term>
<listitem>
<para>
- Invalidates replication slots that are inactive for longer than
- specified amount of time. If this value is specified without units,
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
it is taken as seconds. A value of zero (which is default) disables
the timeout mechanism. This parameter can only be set in
the <filename>postgresql.conf</filename> file or on the server
@@ -4573,11 +4573,9 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</para>
<para>
- This invalidation check happens either when the slot is acquired
- for use or during checkpoint. The time since the slot has become
- inactive is known from its
- <structfield>inactive_since</structfield> value using which the
- timeout is measured.
+ Slot invalidation due to inactivity timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <structfield>inactive_since</structfield> field value.
</para>
<para>
@@ -4585,9 +4583,8 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
applicable for slots on the standby server that are being synced
from primary server (i.e., standby slots having
<structfield>synced</structfield> field <literal>true</literal>).
- Because such synced slots are typically considered not active
- (for them to be later considered as inactive) as they don't perform
- logical decoding to produce the changes.
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index acc0370..bb06592 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -551,12 +551,11 @@ ReplicationSlotName(int index, Name name)
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
*
- * An error is raised if check_for_invalidation is true and the slot has been
+ * An error is raised if error_if_invalid is true and the slot has been
* invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait,
- bool check_for_invalidation)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -635,11 +634,10 @@ retry:
MyReplicationSlot = s;
/*
- * Error out if the slot has been invalidated previously. Because there's
- * no use in acquiring the invalidated slot.
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
*/
- if (check_for_invalidation &&
- s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ if (error_if_invalid && s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
{
Assert(s->inactive_since > 0);
ereport(ERROR,
@@ -1565,6 +1563,7 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
_("You might need to increase \"%s\"."), "max_slot_wal_keep_size");
break;
}
+
case RS_INVAL_HORIZON:
appendStringInfo(&err_detail, _("The slot conflicted with xid horizon %u."),
snapshotConflictHorizon);
@@ -1573,6 +1572,7 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
case RS_INVAL_INACTIVE_TIMEOUT:
Assert(inactive_since > 0);
appendStringInfo(&err_detail,
@@ -1584,6 +1584,7 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
appendStringInfo(err_hint,
_("You might need to increase \"%s\"."), "replication_slot_inactive_timeout");
break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 431cc08..5678e1a 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -253,7 +253,7 @@ extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
extern void ReplicationSlotAcquire(const char *name, bool nowait,
- bool check_for_invalidation);
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
Attachments:
[text/plain] PS_NITPICKS_v440001.txt (4.9K, ../../CAHut+PuC7nwuT3+36zsLxtBAcUHrSaysrHb0_dxTXqMnFBKZBA@mail.gmail.com/2-PS_NITPICKS_v440001.txt)
download | inline diff:
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 970b496..0537714 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4564,8 +4564,8 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</term>
<listitem>
<para>
- Invalidates replication slots that are inactive for longer than
- specified amount of time. If this value is specified without units,
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
it is taken as seconds. A value of zero (which is default) disables
the timeout mechanism. This parameter can only be set in
the <filename>postgresql.conf</filename> file or on the server
@@ -4573,11 +4573,9 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</para>
<para>
- This invalidation check happens either when the slot is acquired
- for use or during checkpoint. The time since the slot has become
- inactive is known from its
- <structfield>inactive_since</structfield> value using which the
- timeout is measured.
+ Slot invalidation due to inactivity timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <structfield>inactive_since</structfield> field value.
</para>
<para>
@@ -4585,9 +4583,8 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
applicable for slots on the standby server that are being synced
from primary server (i.e., standby slots having
<structfield>synced</structfield> field <literal>true</literal>).
- Because such synced slots are typically considered not active
- (for them to be later considered as inactive) as they don't perform
- logical decoding to produce the changes.
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
</para>
</listitem>
</varlistentry>
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index acc0370..bb06592 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -551,12 +551,11 @@ ReplicationSlotName(int index, Name name)
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
*
- * An error is raised if check_for_invalidation is true and the slot has been
+ * An error is raised if error_if_invalid is true and the slot has been
* invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait,
- bool check_for_invalidation)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -635,11 +634,10 @@ retry:
MyReplicationSlot = s;
/*
- * Error out if the slot has been invalidated previously. Because there's
- * no use in acquiring the invalidated slot.
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
*/
- if (check_for_invalidation &&
- s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ if (error_if_invalid && s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
{
Assert(s->inactive_since > 0);
ereport(ERROR,
@@ -1565,6 +1563,7 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
_("You might need to increase \"%s\"."), "max_slot_wal_keep_size");
break;
}
+
case RS_INVAL_HORIZON:
appendStringInfo(&err_detail, _("The slot conflicted with xid horizon %u."),
snapshotConflictHorizon);
@@ -1573,6 +1572,7 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
case RS_INVAL_INACTIVE_TIMEOUT:
Assert(inactive_since > 0);
appendStringInfo(&err_detail,
@@ -1584,6 +1584,7 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
appendStringInfo(err_hint,
_("You might need to increase \"%s\"."), "replication_slot_inactive_timeout");
break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 431cc08..5678e1a 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -253,7 +253,7 @@ extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
extern void ReplicationSlotAcquire(const char *name, bool nowait,
- bool check_for_invalidation);
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
@ 2024-09-08 11:54 ` Bharath Rupireddy <[email protected]>
2024-09-09 05:23 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
0 siblings, 2 replies; 98+ messages in thread
From: Bharath Rupireddy @ 2024-09-08 11:54 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Thanks for reviewing.
On Mon, Sep 2, 2024 at 1:37 PM Peter Smith <[email protected]> wrote:
>
> Commit message.
>
> 1.
> Because such synced slots are typically considered not
> active (for them to be later considered as inactive) as they don't
> perform logical decoding to produce the changes.
>
> This sentence is bad grammar. The docs have the same wording, so
> please see my doc review comment #4 suggestion below.
+1
> 2.
> + <para>
> + Invalidates replication slots that are inactive for longer than
> + specified amount of time. If this value is specified without units,
> + it is taken as seconds. A value of zero (which is default) disables
> + the timeout mechanism. This parameter can only be set in
> + the <filename>postgresql.conf</filename> file or on the server
> + command line.
> + </para>
> +
>
> nit - This is OK as-is, but OTOH why not make the wording consistent
> with the previous GUC description? (e.g. see my v43 [1] #2 review
> comment)
+1.
> 3.
> + <para>
> + This invalidation check happens either when the slot is acquired
> + for use or during checkpoint. The time since the slot has become
> + inactive is known from its
> + <structfield>inactive_since</structfield> value using which the
> + timeout is measured.
> + </para>
> +
>
> I felt this is slightly misleading because slot acquiring has nothing
> to do with setting the slot invalidation anymore. Furthermore, the 2nd
> sentence is bad grammar.
>
> nit - IMO something simple like the following rewording can address
> both of those points:
>
> Slot invalidation due to inactivity timeout occurs during checkpoint.
> The duration of slot inactivity is calculated using the slot's
> <structfield>inactive_since</structfield> field value.
+1.
> 4.
> + Because such synced slots are typically considered not active
> + (for them to be later considered as inactive) as they don't perform
> + logical decoding to produce the changes.
>
> That sentence has bad grammar.
>
> nit – suggest a much simpler replacement:
> Synced slots are always considered to be inactive because they don't
> perform logical decoding to produce changes.
+1.
> 5.
> +#define IsInactiveTimeoutSlotInvalidationApplicable(s) \
>
> 5a.
> I felt this would be better implemented as an inline function. Then it
> can be commented on properly to explain the parts of the condition.
> e.g. the large comment currently in InvalidatePossiblyObsoleteSlot()
> would be more appropriate in this function.
+1.
> 5b.
> The name is very long. Can't it be something shorter/simpler like:
> 'IsSlotATimeoutCandidate()'
>
> ~~~
Missing inactive in the above suggested name. Used
SlotInactiveTimeoutCheckAllowed, similar to XLogInsertAllowed.
> 6. ReplicationSlotAcquire
>
> -ReplicationSlotAcquire(const char *name, bool nowait)
> +ReplicationSlotAcquire(const char *name, bool nowait,
> + bool check_for_invalidation)
>
> nit - Previously this new parameter really did mean to "check" for
> [and set the slot] invalidation. But now I suggest renaming it to
> 'error_if_invalid' to properly reflect the new usage. And also in the
> slot.h.
+1.
> 7.
> + /*
> + * Error out if the slot has been invalidated previously. Because there's
> + * no use in acquiring the invalidated slot.
> + */
>
> nit - The comment is contrary to the code. If there was no reason to
> skip this error, then you would not have the new parameter allowing
> you to skip this error. I suggest just repeating the same comment as
> in the function header.
+1.
> 8. ReportSlotInvalidation
>
> nit - Added some blank lines for consistency.
+1.
> 9. InvalidatePossiblyObsoleteSlot
>
> 9a.
> Consistency is good (commit message, docs and code comments for this),
> but the added sentence has bad grammar. Please see the docs review
> comment #4 above for some alternate phrasing.
+1.
> 9b.
> Now that this logic is moved into a macro (I suggested it should be an
> inline function) IMO this comment does not belong here anymore because
> it is commenting code that you cannot see. Instead, this comment (or
> something like it) should be as comments within the new function.
>
> ======
> src/include/replication/slot.h
+1.
> 10.
> +extern void ReplicationSlotAcquire(const char *name, bool nowait,
> + bool check_for_invalidation);
>
> Change the new param name as described in the earlier review comment.
+1.
> Please refer to the attached file which implements some of the nits
> mentioned above.
Merged the diff into v45. Thanks.
On Tue, Sep 3, 2024 at 12:26 PM Peter Smith <[email protected]> wrote:
>
> TEST CASE #1
>
> 1.
> +# Wait for the inactive replication slot to be invalidated.
>
> Is that comment correct? IIUC the synced slot should *already* be
> invalidated from the primary, so here we are not really "waiting" for
> it to be invalidated; Instead, we are just "confirming" that the
> synchronized slot is already invalidated with the correct reason as
> expected.
Modified the comment.
> 2.
> +# Synced slot mustn't get invalidated on the standby even after a checkpoint,
> +# it must sync invalidation from the primary. So, we must not see the slot's
> +# invalidation message in server log.
>
> This test case seemed bogus, for a couple of reasons:
>
> 2a. IIUC this 'lsub1_sync_slot' is the same one that is already
> invalid (from the primary), so nobody should be surprised that an
> already invalid slot doesn't get flagged as invalid again. i.e.
> Shouldn't your test scenario here be done using a valid synced slot?
+1. Added another test case for checking the synced slot not getting
invalidated despite inactive timeout being set.
> 2b. AFAICT it was only moments above this CHECKPOINT where you
> assigned the standby inactivity timeout to 2s. So even if there was
> some bug invalidating synced slots I don't think you gave it enough
> time to happen -- e.g. I doubt 2s has elapsed yet.
Added sleep(timeout+1) before the checkpoint.
> 3.
> +# Stop standby to make the standby's replication slot on the primary inactive
> +$standby1->stop;
> +
> +# Wait for the standby's replication slot to become inactive
>
> TEST CASE #2
>
> 4.
> +# Stop subscriber to make the replication slot on publisher inactive
> +$subscriber->stop;
> +
> +# Wait for the replication slot to become inactive and then invalidated due to
> +# timeout.
> +wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
> + $inactive_timeout);
>
> IIUC, this is just like comment #3 above. Both these (the stop and the
> wait) seem to belong together, so I think maybe a single bigger
> explanatory comment covering both parts would help for understanding.
Done.
> 5.
> +# Testcase end: Invalidate logical subscriber's slot due to
> +# replication_slot_inactive_timeout.
> +# =============================================================================
>
> IMO the rest of the comment after "Testcase end" isn't very useful.
Removed.
> ======
> sub wait_for_slot_invalidation
>
> 6.
> +sub wait_for_slot_invalidation
> +{
>
> An explanatory header comment for this subroutine would be helpful.
Done.
> 7.
> + # Wait for the replication slot to become inactive
> + $node->poll_query_until(
>
> Why are there are 2 separate poll_query_until's here? Can't those be
> combined into just one?
Ah. My bad. Removed.
> ~~~
>
> 8.
> + # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
> + # for the slot to get invalidated.
> + sleep($inactive_timeout);
> +
>
> Maybe this special sleep to prevent too many CHECKPOINTs should be
> moved to be inside the other subroutine, which is actually doing those
> CHECKPOINTs.
Done.
> 9.
> + # Wait for the inactive replication slot to be invalidated
> + "Timed out while waiting for inactive slot $slot_name to be
> invalidated on node $name";
> +
>
> The comment seems misleading. IIUC you are not "waiting" for the
> invalidation here, because it is the other subroutine doing the
> waiting for the invalidation message in the logs. Instead, here I
> think you are just confirming the 'invalidation_reason' got set
> correctly. The comment should say what it is really doing.
Modified.
> sub check_for_slot_invalidation_in_server_log
>
> 10.
> +# Check for invalidation of slot in server log
> +sub check_for_slot_invalidation_in_server_log
> +{
>
> I think the main function of this subroutine is the CHECKPOINT and the
> waiting for the server log to say invalidation happened. It is doing a
> loop of a) CHECKPOINT then b) inspecting the server log for the slot
> invalidation, and c) waiting for a bit. Repeat 10 times.
>
> A comment describing the logic for this subroutine would be helpful.
>
> The most important side-effect of this function is the CHECKPOINT
> because without that nothing will ever get invalidated due to
> inactivity, but this key point is not obvious from the subroutine
> name.
>
> IMO it would be better to name this differently to reflect what it is
> really doing:
> e.g. "CHECKPOINT_and_wait_for_slot_invalidation_in_server_log"
That would be too long. Changed the function name to
trigger_slot_invalidation() which is appropriate.
Please find the v45 patch. Addressed above and Shveta's review comments [1].
Amit's comments [2] and [3] are still pending.
[1] https://www.postgresql.org/message-id/CAJpy0uC8Dg-0JS3NRUwVUemgz5Ar2v3_EQQFXyAigWSEQ8U47Q%40mail.gma...
[2] https://www.postgresql.org/message-id/CAA4eK1K7DdT_5HnOWs5tVPYC%3D-h%2Bm85wu7k-7RVJaJ7zMxprWQ%40mail...
[3] https://www.postgresql.org/message-id/CAA4eK1%2Bkt-QRr1RP%3DD%3D4_tp%2BS%2BCErQ6rNe7KVYEyZ3f6PYXpvw%...
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v45-0001-Add-inactive_timeout-based-replication-slot-inva.patch (34.5K, ../../CALj2ACWXQT3_HY40ceqKf1DadjLQP6b1r=0sZRh-xhAOd-b0pA@mail.gmail.com/2-v45-0001-Add-inactive_timeout-based-replication-slot-inva.patch)
download | inline diff:
From 60a5beb355ef2c9c2392e0229af0a24293882f50 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Sun, 8 Sep 2024 11:33:05 +0000
Subject: [PATCH v45] Add inactive_timeout based replication slot invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage for instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named replication_slot_inactive_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are inactive for longer than this amount of
time.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila
Reviewed-by: Ajin Cherian, Shveta Malik, Peter Smith
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/202403260841.5jcv7ihniccy%40alvherre.pgsql
---
doc/src/sgml/config.sgml | 35 ++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 8 +-
src/backend/replication/slot.c | 171 ++++++++--
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 24 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 321 ++++++++++++++++++
13 files changed, 560 insertions(+), 30 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 0aec11f443..27b2285da1 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4556,6 +4556,41 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the inactive timeout invalidation mechanism. This parameter can only
+ be set in the <filename>postgresql.conf</filename> file or on the
+ server command line.
+ </para>
+
+ <para>
+ Slot invalidation due to inactivity timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from primary server (i.e., standby slots having
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+ value <literal>true</literal>).
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 634a4c0fab..5633429eef 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2618,6 +2618,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for longer than the amount of time specified by the
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index f9649eec1a..6cc7a739ec 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -448,7 +448,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -667,7 +667,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
@@ -1543,9 +1543,7 @@ update_synced_slots_inactive_since(void)
if (now == 0)
now = GetCurrentTimestamp();
- SpinLockAcquire(&s->mutex);
- s->inactive_since = now;
- SpinLockRelease(&s->mutex);
+ ReplicationSlotSetInactiveSince(s, &now, true);
}
}
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 0a03776156..d92b92bfed 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -159,6 +161,14 @@ static XLogRecPtr ss_oldest_flush_lsn = InvalidXLogRecPtr;
static void ReplicationSlotShmemExit(int code, Datum arg);
static void ReplicationSlotDropPtr(ReplicationSlot *slot);
+static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
+ ReplicationSlot *s,
+ XLogRecPtr oldestLSN,
+ Oid dboid,
+ TransactionId snapshotConflictHorizon,
+ bool *invalidated);
+static inline bool SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s);
+
/* internal persistency functions */
static void RestoreSlotFromDisk(const char *name);
static void CreateSlotOnDisk(ReplicationSlot *slot);
@@ -535,9 +545,12 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +628,21 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
+ */
+ if (error_if_invalid && s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive for longer than the amount of time specified by \"%s\".",
+ "replication_slot_inactive_timeout.")));
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -703,16 +731,12 @@ ReplicationSlotRelease(void)
*/
SpinLockAcquire(&slot->mutex);
slot->active_pid = 0;
- slot->inactive_since = now;
+ ReplicationSlotSetInactiveSince(slot, &now, false);
SpinLockRelease(&slot->mutex);
ConditionVariableBroadcast(&slot->active_cv);
}
else
- {
- SpinLockAcquire(&slot->mutex);
- slot->inactive_since = now;
- SpinLockRelease(&slot->mutex);
- }
+ ReplicationSlotSetInactiveSince(slot, &now, true);
MyReplicationSlot = NULL;
@@ -785,7 +809,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +836,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, true);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1501,7 +1525,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1531,6 +1556,15 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for longer than the amount of time specified by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1547,6 +1581,28 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
pfree(err_detail.data);
}
+/*
+ * Is this replication slot allowed for inactive timeout invalidation check?
+ *
+ * Check if inactive timeout invalidation mechanism is disabled or slot is
+ * currently being used or server is in recovery mode or slot on standby is
+ * currently being synced from the primary.
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s)
+{
+ return (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !RecoveryInProgress() &&
+ !s->data.synced);
+}
+
/*
* Helper for InvalidateObsoleteReplicationSlots
*
@@ -1574,6 +1630,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1581,6 +1638,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1591,6 +1649,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ SlotInactiveTimeoutCheckAllowed(s))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1644,6 +1712,28 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ if (!SlotInactiveTimeoutCheckAllowed(s))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1669,11 +1759,13 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1728,7 +1820,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1774,7 +1867,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1797,6 +1891,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1849,7 +1944,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1907,6 +2003,38 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
@@ -2201,6 +2329,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now = 0;
/* no need to lock here, no concurrent access allowed yet */
@@ -2388,12 +2517,16 @@ RestoreSlotFromDisk(const char *name)
slot->in_use = true;
slot->active_pid = 0;
+ /* Use the same inactive_since time for all the slots. */
+ if (now == 0)
+ now = GetCurrentTimestamp();
+
/*
* Set the time since the slot has become inactive after loading the
* slot from the disk into memory. Whoever acquires the slot i.e.
* makes the slot active will reset it.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ ReplicationSlotSetInactiveSince(slot, &now, false);
restored = true;
break;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index c7bfbb15e0..b1b7b075bd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -540,7 +540,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c5f1009f37..61a0e38715 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -844,7 +844,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1462,7 +1462,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 686309db58..5e27cd3270 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a..deca3a4aeb 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 45582cf9d8..27c4f107e5 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -224,6 +226,24 @@ typedef struct ReplicationSlotCtlData
ReplicationSlot replication_slots[1];
} ReplicationSlotCtlData;
+/*
+ * Set slot's inactive_since property unless it was previously invalidated due
+ * to inactive timeout.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz *now,
+ bool acquire_lock)
+{
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ if (s->data.invalidated != RS_INVAL_INACTIVE_TIMEOUT)
+ s->inactive_since = *now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
+
/*
* Pointers to shared memory
*/
@@ -233,6 +253,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -249,7 +270,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 712924c2fa..c45a5106f4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -52,6 +52,7 @@ tests += {
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
't/043_wal_replay_wait.pl',
+ 't/050_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..669d6ccc7a
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,321 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start
+#
+# Invalidate streaming standby slot and logical failover slot on primary due to
+# inactive timeout. Also, check logical failover slot synced to standby from
+# primary doesn't invalidate on its own, but gets the invalidated state from the
+# primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot1'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot1', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+# Sync primary slot to standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that logical failover slot is created on standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot sync_slot1 has synced as true on standby');
+
+my $logstart = -s $primary->logfile;
+my $inactive_timeout = 1;
+
+# Set timeout so that next checkpoint will invalidate inactive slot
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Check for logical failover slot to become inactive on primary. Note that
+# nobody has acquired slot yet, so it must get invalidated due to
+# inactive timeout.
+check_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+ $inactive_timeout);
+
+# Sync primary slot to standby. Note that primary slot has already been
+# invalidated due to inactive timeout. Standby must just sync inavalidated
+# state.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for sync_slot1 invalidation to be synced on standby";
+
+# Make standby slot on primary inactive and check for invalidation
+$standby1->stop;
+check_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# =============================================================================
+# Testcase start
+# Synced slot mustn't get invalidated on standby on its own due to inactive
+# timeout.
+
+# Disable inactive timeout on primary
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$primary->reload;
+
+# Create standby
+my $standby2 = PostgreSQL::Test::Cluster->new('standby2');
+$standby2->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+$connstr = $primary->connstr;
+$standby2->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot2'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot2', 'test_decoding', false, false, true);}
+);
+
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot2', immediately_reserve := true);
+]);
+
+$standby2->start;
+
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby2);
+
+
+$standby2->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$standby2->reload;
+
+# Sync primary slot to standby
+$standby2->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that logical failover slot is created on standby
+is( $standby2->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot2' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot sync_slot2 has synced as true on standby');
+
+$logstart = -s $standby2->logfile;
+
+# Give enough time
+sleep($inactive_timeout+1);
+
+# Despite inactive timeout being set, synced slot won't get invalidated on its
+# own on standby. So, we must not see invalidation message in server log.
+$standby2->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby2->log_contains(
+ "invalidating obsolete replication slot \"sync_slot2\"",
+ $logstart),
+ 'check that syned sync_slot2 has not been invalidated on the standby'
+);
+
+$standby2->stop;
+
+# Testcase end
+# =============================================================================
+
+# =============================================================================
+# Testcase start
+# Invalidate logical subscriber slot due to inactive timeout.
+
+my $publisher = $primary;
+
+# Prepare for test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+
+is($result, qq(5), "check initial copy was done");
+
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Make subscriber slot on publisher inactive and check for invalidation
+$subscriber->stop;
+check_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# Check for slot to first become inactive and then get invalidated
+sub check_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+
+ # Wait for slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND active = 'f' AND
+ inactive_since IS NOT NULL;
+ ])
+ or die
+ "Timed out while waiting for slot $slot to become inactive on node $name";
+
+ trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout);
+
+ # Wait for invalidation reason to be set
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $name";
+
+ # Check that invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
+
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot"/,
+ "detected error upon trying to acquire invalidated slot $slot on node $name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot on node $name";
+}
+
+# Trigger slot invalidation and confirm it in server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+ my $invalidated = 0;
+
+ # Give enough time to avoid multiple checkpoints
+ sleep($inactive_timeout+1);
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot invalidation has been logged on node $name"
+ );
+}
+
+done_testing();
--
2.43.0
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-09 05:23 ` shveta malik <[email protected]>
2024-09-16 09:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: shveta malik @ 2024-09-09 05:23 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Sun, Sep 8, 2024 at 5:25 PM Bharath Rupireddy
<[email protected]> wrote:
>
>
> Please find the v45 patch. Addressed above and Shveta's review comments [1].
>
Thanks for the patch. Please find my comments:
1)
src/sgml/config.sgml:
+ Synced slots are always considered to be inactive because they
don't perform logical decoding to produce changes.
It is better we avoid such a statement, as internally we use logical
decoding to advance restart-lsn, see
'LogicalSlotAdvanceAndCheckSnapState' called form slotsync.c.
<Also see related comment 6 below>
2)
src/sgml/config.sgml:
+ disables the inactive timeout invalidation mechanism
+ Slot invalidation due to inactivity timeout occurs during checkpoint.
Either have 'inactive' at both the places or 'inactivity'.
3)
slot.c:
+static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause
cause,
+ ReplicationSlot *s,
+ XLogRecPtr oldestLSN,
+ Oid dboid,
+ TransactionId snapshotConflictHorizon,
+ bool *invalidated);
+static inline bool SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s);
I think, we do not need above 2 declarations. The code compile fine
without these as the usage is later than the definition.
4)
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
+ */
+ if (error_if_invalid && s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
The comment is generic while the 'if condition' is specific to one
invalidation cause. Even though I feel it can be made generic test for
all invalidation causes but that is not under scope of this thread and
needs more testing/analysis. For the time being, we can make comment
specific to the concerned invalidation cause. The header of function
will also need the same change.
5)
SlotInactiveTimeoutCheckAllowed():
+ * Check if inactive timeout invalidation mechanism is disabled or slot is
+ * currently being used or server is in recovery mode or slot on standby is
+ * currently being synced from the primary.
+ *
These comments say exact opposite of what we are checking in code.
Since the function name has 'Allowed' in it, we should be putting
comments which say what allows it instead of what disallows it.
6)
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s)
Perhaps we should avoid mentioning logical decoding here. When slots
are synced, they are performing decoding and their inactive_since is
changing continuously. A better way to make this statement will be:
We want to ensure that the slots being synchronized are not
invalidated, as they need to be preserved for future use when the
standby server is promoted to the primary. This is necessary for
resuming logical replication from the new primary server.
<Rephrase if needed>
7)
InvalidatePossiblyObsoleteSlot()
we are calling SlotInactiveTimeoutCheckAllowed() twice in this
function. We shall optimize.
At the first usage place, shall we simply get timestamp when cause is
RS_INVAL_INACTIVE_TIMEOUT without checking
SlotInactiveTimeoutCheckAllowed() as IMO it does not seem a
performance critical section. Or if we retain check at first place,
then at the second place we can avoid calling it again based on
whether 'now' is NULL or not.
thanks
Shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 05:23 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-16 09:47 ` Bharath Rupireddy <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Bharath Rupireddy @ 2024-09-16 09:47 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Thanks for reviewing.
On Mon, Sep 9, 2024 at 10:54 AM shveta malik <[email protected]> wrote:
>
> 2)
> src/sgml/config.sgml:
>
> + disables the inactive timeout invalidation mechanism
>
> + Slot invalidation due to inactivity timeout occurs during checkpoint.
>
> Either have 'inactive' at both the places or 'inactivity'.
Used "inactive timeout".
> 3)
> slot.c:
> +static bool InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause
> cause,
> + ReplicationSlot *s,
> + XLogRecPtr oldestLSN,
> + Oid dboid,
> + TransactionId snapshotConflictHorizon,
> + bool *invalidated);
> +static inline bool SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s);
>
> I think, we do not need above 2 declarations. The code compile fine
> without these as the usage is later than the definition.
Hm, it's a usual practice that I follow irrespective of the placement
of function declarations. Since it was brought up, I removed the
declarations.
> 4)
> + /*
> + * An error is raised if error_if_invalid is true and the slot has been
> + * invalidated previously.
> + */
> + if (error_if_invalid && s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
>
> The comment is generic while the 'if condition' is specific to one
> invalidation cause. Even though I feel it can be made generic test for
> all invalidation causes but that is not under scope of this thread and
> needs more testing/analysis.
Right.
> For the time being, we can make comment
> specific to the concerned invalidation cause. The header of function
> will also need the same change.
Adjusted the comment, but left the variable name error_if_invalid as
is. Didn't want to make it long, one can look at the code to
understand what it is used for.
> 5)
> SlotInactiveTimeoutCheckAllowed():
>
> + * Check if inactive timeout invalidation mechanism is disabled or slot is
> + * currently being used or server is in recovery mode or slot on standby is
> + * currently being synced from the primary.
> + *
>
> These comments say exact opposite of what we are checking in code.
> Since the function name has 'Allowed' in it, we should be putting
> comments which say what allows it instead of what disallows it.
Modified.
> 1)
> src/sgml/config.sgml:
>
> + Synced slots are always considered to be inactive because they
> don't perform logical decoding to produce changes.
>
> It is better we avoid such a statement, as internally we use logical
> decoding to advance restart-lsn, see
> 'LogicalSlotAdvanceAndCheckSnapState' called form slotsync.c.
> <Also see related comment 6 below>
>
> 6)
>
> + * Synced slots are always considered to be inactive because they don't
> + * perform logical decoding to produce changes.
> + */
> +static inline bool
> +SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s)
>
> Perhaps we should avoid mentioning logical decoding here. When slots
> are synced, they are performing decoding and their inactive_since is
> changing continuously. A better way to make this statement will be:
>
> We want to ensure that the slots being synchronized are not
> invalidated, as they need to be preserved for future use when the
> standby server is promoted to the primary. This is necessary for
> resuming logical replication from the new primary server.
> <Rephrase if needed>
They are performing logical decoding, but not producing the changes
for the clients to consume. So, IMO, the accompanying "to produce
changes" next to the "logical decoding" is good here.
> 7)
>
> InvalidatePossiblyObsoleteSlot()
>
> we are calling SlotInactiveTimeoutCheckAllowed() twice in this
> function. We shall optimize.
>
> At the first usage place, shall we simply get timestamp when cause is
> RS_INVAL_INACTIVE_TIMEOUT without checking
> SlotInactiveTimeoutCheckAllowed() as IMO it does not seem a
> performance critical section. Or if we retain check at first place,
> then at the second place we can avoid calling it again based on
> whether 'now' is NULL or not.
Getting a current timestamp can get costlier on platforms that use
various clock sources, so assigning 'now' unconditionally isn't the
way IMO. Using the inline function in two places improves the
readability. Can optimize it if there's any performance impact of
calling the inline function in two places.
Will post the new patch version soon.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-09 07:40 ` Peter Smith <[email protected]>
2024-09-10 01:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
1 sibling, 2 replies; 98+ messages in thread
From: Peter Smith @ 2024-09-09 07:40 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi, here are some review comments for v45-0001 (excluding the test code)
======
doc/src/sgml/config.sgml
1.
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from primary server (i.e., standby slots having
nit - /from primary server/from the primary server/
======
src/backend/replication/slot.c
2. ReplicationSlotAcquire
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive
for longer than the amount of time specified by \"%s\".",
+ "replication_slot_inactive_timeout.")));
nit - "replication_slot_inactive_timeout." - should be no period
inside that GUC name literal
~~~
3. ReportSlotInvalidation
I didn't understand why there was a hint for:
"You might need to increase \"%s\".", "max_slot_wal_keep_size"
But you don't have an equivalent hint for timeout invalidation:
"You might need to increase \"%s\".", "replication_slot_inactive_timeout"
Why aren't these similar cases consistent?
~~~
4. RestoreSlotFromDisk
+ /* Use the same inactive_since time for all the slots. */
+ if (now == 0)
+ now = GetCurrentTimestamp();
+
Is the deferred assignment really necessary? Why not just
unconditionally assign the 'now' just before the for-loop? Or even at
the declaration? e.g. The 'replication_slot_inactive_timeout' is
measured in seconds so I don't think 'inactive_since' being wrong by a
millisecond here will make any difference.
======
src/include/replication/slot.h
5. ReplicationSlotSetInactiveSince
+/*
+ * Set slot's inactive_since property unless it was previously invalidated due
+ * to inactive timeout.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz *now,
+ bool acquire_lock)
+{
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ if (s->data.invalidated != RS_INVAL_INACTIVE_TIMEOUT)
+ s->inactive_since = *now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
Is the logic correct? What if the slot was already invalid due to some
reason other than RS_INVAL_INACTIVE_TIMEOUT? Is an Assert needed?
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 27b2285..97b4fb5 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4582,7 +4582,7 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
<para>
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
- from primary server (i.e., standby slots having
+ from the primary server (i.e., standby slots having
<link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
value <literal>true</literal>).
Synced slots are always considered to be inactive because they don't
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index d92b92b..8cc67b4 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -640,7 +640,7 @@ retry:
errmsg("can no longer get changes from replication slot \"%s\"",
NameStr(s->data.name)),
errdetail("This slot has been invalidated because it was inactive for longer than the amount of time specified by \"%s\".",
- "replication_slot_inactive_timeout.")));
+ "replication_slot_inactive_timeout")));
}
/*
Attachments:
[text/plain] PS_NITPICKS_20240909_TIMEOUT_V450001.txt (1.3K, ../../CAHut+PtJxxaVhjDCFPqE_V63VaAN_bL+PHRMak=dkghoGrZc0Q@mail.gmail.com/2-PS_NITPICKS_20240909_TIMEOUT_V450001.txt)
download | inline diff:
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 27b2285..97b4fb5 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4582,7 +4582,7 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
<para>
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
- from primary server (i.e., standby slots having
+ from the primary server (i.e., standby slots having
<link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
value <literal>true</literal>).
Synced slots are always considered to be inactive because they don't
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index d92b92b..8cc67b4 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -640,7 +640,7 @@ retry:
errmsg("can no longer get changes from replication slot \"%s\"",
NameStr(s->data.name)),
errdetail("This slot has been invalidated because it was inactive for longer than the amount of time specified by \"%s\".",
- "replication_slot_inactive_timeout.")));
+ "replication_slot_inactive_timeout")));
}
/*
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
@ 2024-09-10 01:34 ` Peter Smith <[email protected]>
1 sibling, 0 replies; 98+ messages in thread
From: Peter Smith @ 2024-09-10 01:34 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi, here is the remainder of my v45-0001 review. These comments are
for the test code only.
======
Testcase #1
1.
+# Testcase start
+#
+# Invalidate streaming standby slot and logical failover slot on primary due to
+# inactive timeout. Also, check logical failover slot synced to standby from
+# primary doesn't invalidate on its own, but gets the invalidated
state from the
+# primary.
nit - s/primary/the primary/ (in a couple of places)
nit - s/standby/the standby/
nit - other trivial tweaks.
~~~
2.
+# Create sync slot on primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1',
'test_decoding', false, false, true);}
+);
nit - s/primary/the primary/
~~~
3.
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name :=
'sb_slot1', immediately_reserve := true);
+]);
Should this have a comment?
~~~
4.
+# Wait until standby has replayed enough data
+$primary->wait_for_catchup($standby1);
nit - s/standby/the standby/
~~~
5.
+# Sync primary slot to standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
nit - /Sync primary slot to standby/Sync the primary slots to the standby/
~~~
6.
+# Confirm that logical failover slot is created on standby
nit - s/Confirm that logical failover slot is created on
standby/Confirm that the logical failover slot is created on the
standby/
~~~
7.
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced AND NOT temporary;}
+ ),
+ "t",
+ 'logical slot sync_slot1 has synced as true on standby');
IMO here you should also be checking that the sync slot state is NOT
invalidated, just as a counterpoint for the test part later that
checks that it IS invalidated.
~~~
8.
+my $inactive_timeout = 1;
+
+# Set timeout so that next checkpoint will invalidate inactive slot
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO
'${inactive_timeout}s';
+]);
+$primary->reload;
8a.
nit - I think that $inactive_timeout assignment belongs below your comment.
~
8b.
nit - s/Set timeout so that next checkpoint will invalidate inactive
slot/Set timeout GUC so that the next checkpoint will invalidate
inactive slots/
~~~
9.
+# Check for logical failover slot to become inactive on primary. Note that
+# nobody has acquired slot yet, so it must get invalidated due to
+# inactive timeout.
nit - /Check for logical failover slot to become inactive on
primary./Wait for logical failover slot to become inactive on the
primary./
nit - /has acquired slot/has acquired the slot/
~~~
10.
+# Sync primary slot to standby. Note that primary slot has already been
+# invalidated due to inactive timeout. Standby must just sync inavalidated
+# state.
nit - minor, add "the". fix typo "inavalidated", etc. suggestion:
Re-sync the primary slots to the standby. Note that the primary slot was already
invalidated (above) due to inactive timeout. The standby must just
sync the invalidated
state.
~~~
11.
+# Make standby slot on primary inactive and check for invalidation
+$standby1->stop;
nit - /standby slot/the standby slot/
nit - /on primary/on the primary/
======
Testcase #2
12.
I'm not sure it is necessary to do all this extra work. IIUC, there
was already almost everything you needed in the previous Testcase #1.
So, I thought you could just combine this extra standby timeout test
in Testcase #1.
Indeed, your Testcase #1 comment still says it is doing this: ("Also,
check logical failover slot synced to standby from primary doesn't
invalidate on its own,...")
e.g.
- NEW: set the GUC timeout on the standby
- sync the sync_slot (already doing in test #1)
- ensure the synced slot is NOT invalid (already suggested above for test #1)
- NEW: then do a standby sleep > timeout duration
- NEW: then do a standby CHECKPOINT...
- NEW: then ensure the sync slot invalidation did NOT happen
- then proceed with the rest of test #1...
======
Testcase #3
13.
nit - remove a few blank lines to group associated statements together.
~~~
14.
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '
${inactive_timeout}s';
+]);
+$publisher->reload;
nit - this deserves a comment, the same as in Testcase #1
======
sub wait_for_slot_invalidation
15.
+# Check for slot to first become inactive and then get invalidated
+sub check_for_slot_invalidation
nit - IMO the previous name was better (e.g. "wait_for.." instead of
"check_for...") because that describes exactly what the subroutine is
doing.
suggestion:
# Wait for the slot to first become inactive and then get invalidated
sub wait_for_slot_invalidation
~~~
16.
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
The variable $name seems too vague. How about $node_name?
~~~
17.
+ # Wait for invalidation reason to be set
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to
be set on node $name";
17a.
nit - /# Wait for invalidation reason to be set/# Check that the
invalidation reason is 'inactive_timeout'/
IIUC, the 'trigger_slot_invalidation' function has already invalidated
the slot at this point, so we are not really "Waiting..."; we are
"Checking..." that the reason was correctly set.
~
17b.
I think this code fragment maybe would be better put inside the
'trigger_slot_invalidation' function. (I've done this in the nitpicks
attachment)
~~~
18.
+ # Check that invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
18a.
s/Check that invalidated slot/Check that an invalidated slot/
~
18b.
nit - Remove some blank lines, because the comment applies to all below it.
======
sub trigger_slot_invalidation
19.
+# Trigger slot invalidation and confirm it in server log
+sub trigger_slot_invalidation
nit - s/confirm it in server log/confirm it in the server log/
~
20.
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $name = $node->name;
+ my $invalidated = 0;
(same as the other subroutine)
nit - The variable $name seems too vague. How about $node_name?
======
Please refer to the attached nitpicks top-up patch which implements
most of the above nits.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
index 669d6cc..34b46d5 100644
--- a/src/test/recovery/t/050_invalidate_slots.pl
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -12,10 +12,10 @@ use Time::HiRes qw(usleep);
# =============================================================================
# Testcase start
#
-# Invalidate streaming standby slot and logical failover slot on primary due to
-# inactive timeout. Also, check logical failover slot synced to standby from
-# primary doesn't invalidate on its own, but gets the invalidated state from the
-# primary.
+# Invalidate a streaming standby slot and logical failover slot on the primary
+# due to inactive timeout. Also, check that a logical failover slot synced to
+# the standby from the primary doesn't invalidate on its own, but gets the
+# invalidated state from the primary.
# Initialize primary
my $primary = PostgreSQL::Test::Cluster->new('primary');
@@ -45,7 +45,7 @@ primary_slot_name = 'sb_slot1'
primary_conninfo = '$connstr dbname=postgres'
));
-# Create sync slot on primary
+# Create sync slot on the primary
$primary->psql('postgres',
q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
);
@@ -57,13 +57,13 @@ $primary->safe_psql(
$standby1->start;
-# Wait until standby has replayed enough data
+# Wait until the standby has replayed enough data
$primary->wait_for_catchup($standby1);
-# Sync primary slot to standby
+# Sync the primary slots to the standby
$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
-# Confirm that logical failover slot is created on standby
+# Confirm that the logical failover slot is created on the standby
is( $standby1->safe_psql(
'postgres',
q{SELECT count(*) = 1 FROM pg_replication_slots
@@ -73,24 +73,24 @@ is( $standby1->safe_psql(
'logical slot sync_slot1 has synced as true on standby');
my $logstart = -s $primary->logfile;
-my $inactive_timeout = 1;
-# Set timeout so that next checkpoint will invalidate inactive slot
+# Set timeout GUC so that that next checkpoint will invalidate inactive slots
+my $inactive_timeout = 1;
$primary->safe_psql(
'postgres', qq[
ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
]);
$primary->reload;
-# Check for logical failover slot to become inactive on primary. Note that
+# Wait for logical failover slot to become inactive on the primary. Note that
# nobody has acquired slot yet, so it must get invalidated due to
# inactive timeout.
-check_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
$inactive_timeout);
-# Sync primary slot to standby. Note that primary slot has already been
-# invalidated due to inactive timeout. Standby must just sync inavalidated
-# state.
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to inactive timeout. The standby must just
+# sync the invalidated state.
$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
$standby1->poll_query_until(
'postgres', qq[
@@ -101,9 +101,9 @@ $standby1->poll_query_until(
or die
"Timed out while waiting for sync_slot1 invalidation to be synced on standby";
-# Make standby slot on primary inactive and check for invalidation
+# Make the standby slot on the primary inactive and check for invalidation
$standby1->stop;
-check_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
$inactive_timeout);
# Testcase end
@@ -223,14 +223,12 @@ $publisher->safe_psql(
$subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
);
-
$subscriber->wait_for_subscription_sync($publisher, 'sub');
-
my $result =
$subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
-
is($result, qq(5), "check initial copy was done");
+# Set timeout GUC so that that next checkpoint will invalidate inactive slots
$publisher->safe_psql(
'postgres', qq[
ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
@@ -241,17 +239,17 @@ $logstart = -s $publisher->logfile;
# Make subscriber slot on publisher inactive and check for invalidation
$subscriber->stop;
-check_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
$inactive_timeout);
# Testcase end
# =============================================================================
-# Check for slot to first become inactive and then get invalidated
-sub check_for_slot_invalidation
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
{
my ($node, $slot, $offset, $inactive_timeout) = @_;
- my $name = $node->name;
+ my $node_name = $node->name;
# Wait for slot to become inactive
$node->poll_query_until(
@@ -261,45 +259,33 @@ sub check_for_slot_invalidation
inactive_since IS NOT NULL;
])
or die
- "Timed out while waiting for slot $slot to become inactive on node $name";
+ "Timed out while waiting for slot $slot to become inactive on node $node_name";
trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout);
- # Wait for invalidation reason to be set
- $node->poll_query_until(
- 'postgres', qq[
- SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
- WHERE slot_name = '$slot' AND
- invalidation_reason = 'inactive_timeout';
- ])
- or die
- "Timed out while waiting for invalidation reason of slot $slot to be set on node $name";
-
- # Check that invalidated slot cannot be acquired
+ # Check that an invalidated slot cannot be acquired
my ($result, $stdout, $stderr);
-
($result, $stdout, $stderr) = $node->psql(
'postgres', qq[
SELECT pg_replication_slot_advance('$slot', '0/1');
]);
-
ok( $stderr =~
/can no longer get changes from replication slot "$slot"/,
- "detected error upon trying to acquire invalidated slot $slot on node $name"
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
)
or die
- "could not detect error upon trying to acquire invalidated slot $slot on node $name";
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
}
-# Trigger slot invalidation and confirm it in server log
+# Trigger slot invalidation and confirm it in the server log
sub trigger_slot_invalidation
{
my ($node, $slot, $offset, $inactive_timeout) = @_;
- my $name = $node->name;
+ my $node_name = $node->name;
my $invalidated = 0;
# Give enough time to avoid multiple checkpoints
- sleep($inactive_timeout+1);
+ sleep($inactive_timeout + 1);
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
@@ -314,8 +300,18 @@ sub trigger_slot_invalidation
usleep(100_000);
}
ok($invalidated,
- "check that slot $slot invalidation has been logged on node $name"
+ "check that slot $slot invalidation has been logged on node $node_name"
);
+
+ # Check that the invalidation reason is 'inactive_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
}
done_testing();
Attachments:
[text/plain] PS_NITPICKS_20240910_SLOT_V450001_TESTS.txt (7.4K, ../../CAHut+PsHmfuDQt_gN2udjb9Ln00EvWw-eiRag5prB7JOgn9eEA@mail.gmail.com/2-PS_NITPICKS_20240910_SLOT_V450001_TESTS.txt)
download | inline diff:
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
index 669d6cc..34b46d5 100644
--- a/src/test/recovery/t/050_invalidate_slots.pl
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -12,10 +12,10 @@ use Time::HiRes qw(usleep);
# =============================================================================
# Testcase start
#
-# Invalidate streaming standby slot and logical failover slot on primary due to
-# inactive timeout. Also, check logical failover slot synced to standby from
-# primary doesn't invalidate on its own, but gets the invalidated state from the
-# primary.
+# Invalidate a streaming standby slot and logical failover slot on the primary
+# due to inactive timeout. Also, check that a logical failover slot synced to
+# the standby from the primary doesn't invalidate on its own, but gets the
+# invalidated state from the primary.
# Initialize primary
my $primary = PostgreSQL::Test::Cluster->new('primary');
@@ -45,7 +45,7 @@ primary_slot_name = 'sb_slot1'
primary_conninfo = '$connstr dbname=postgres'
));
-# Create sync slot on primary
+# Create sync slot on the primary
$primary->psql('postgres',
q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
);
@@ -57,13 +57,13 @@ $primary->safe_psql(
$standby1->start;
-# Wait until standby has replayed enough data
+# Wait until the standby has replayed enough data
$primary->wait_for_catchup($standby1);
-# Sync primary slot to standby
+# Sync the primary slots to the standby
$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
-# Confirm that logical failover slot is created on standby
+# Confirm that the logical failover slot is created on the standby
is( $standby1->safe_psql(
'postgres',
q{SELECT count(*) = 1 FROM pg_replication_slots
@@ -73,24 +73,24 @@ is( $standby1->safe_psql(
'logical slot sync_slot1 has synced as true on standby');
my $logstart = -s $primary->logfile;
-my $inactive_timeout = 1;
-# Set timeout so that next checkpoint will invalidate inactive slot
+# Set timeout GUC so that that next checkpoint will invalidate inactive slots
+my $inactive_timeout = 1;
$primary->safe_psql(
'postgres', qq[
ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
]);
$primary->reload;
-# Check for logical failover slot to become inactive on primary. Note that
+# Wait for logical failover slot to become inactive on the primary. Note that
# nobody has acquired slot yet, so it must get invalidated due to
# inactive timeout.
-check_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
$inactive_timeout);
-# Sync primary slot to standby. Note that primary slot has already been
-# invalidated due to inactive timeout. Standby must just sync inavalidated
-# state.
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to inactive timeout. The standby must just
+# sync the invalidated state.
$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
$standby1->poll_query_until(
'postgres', qq[
@@ -101,9 +101,9 @@ $standby1->poll_query_until(
or die
"Timed out while waiting for sync_slot1 invalidation to be synced on standby";
-# Make standby slot on primary inactive and check for invalidation
+# Make the standby slot on the primary inactive and check for invalidation
$standby1->stop;
-check_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
$inactive_timeout);
# Testcase end
@@ -223,14 +223,12 @@ $publisher->safe_psql(
$subscriber->safe_psql('postgres',
"CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
);
-
$subscriber->wait_for_subscription_sync($publisher, 'sub');
-
my $result =
$subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
-
is($result, qq(5), "check initial copy was done");
+# Set timeout GUC so that that next checkpoint will invalidate inactive slots
$publisher->safe_psql(
'postgres', qq[
ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
@@ -241,17 +239,17 @@ $logstart = -s $publisher->logfile;
# Make subscriber slot on publisher inactive and check for invalidation
$subscriber->stop;
-check_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
$inactive_timeout);
# Testcase end
# =============================================================================
-# Check for slot to first become inactive and then get invalidated
-sub check_for_slot_invalidation
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
{
my ($node, $slot, $offset, $inactive_timeout) = @_;
- my $name = $node->name;
+ my $node_name = $node->name;
# Wait for slot to become inactive
$node->poll_query_until(
@@ -261,45 +259,33 @@ sub check_for_slot_invalidation
inactive_since IS NOT NULL;
])
or die
- "Timed out while waiting for slot $slot to become inactive on node $name";
+ "Timed out while waiting for slot $slot to become inactive on node $node_name";
trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout);
- # Wait for invalidation reason to be set
- $node->poll_query_until(
- 'postgres', qq[
- SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
- WHERE slot_name = '$slot' AND
- invalidation_reason = 'inactive_timeout';
- ])
- or die
- "Timed out while waiting for invalidation reason of slot $slot to be set on node $name";
-
- # Check that invalidated slot cannot be acquired
+ # Check that an invalidated slot cannot be acquired
my ($result, $stdout, $stderr);
-
($result, $stdout, $stderr) = $node->psql(
'postgres', qq[
SELECT pg_replication_slot_advance('$slot', '0/1');
]);
-
ok( $stderr =~
/can no longer get changes from replication slot "$slot"/,
- "detected error upon trying to acquire invalidated slot $slot on node $name"
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
)
or die
- "could not detect error upon trying to acquire invalidated slot $slot on node $name";
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
}
-# Trigger slot invalidation and confirm it in server log
+# Trigger slot invalidation and confirm it in the server log
sub trigger_slot_invalidation
{
my ($node, $slot, $offset, $inactive_timeout) = @_;
- my $name = $node->name;
+ my $node_name = $node->name;
my $invalidated = 0;
# Give enough time to avoid multiple checkpoints
- sleep($inactive_timeout+1);
+ sleep($inactive_timeout + 1);
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
@@ -314,8 +300,18 @@ sub trigger_slot_invalidation
usleep(100_000);
}
ok($invalidated,
- "check that slot $slot invalidation has been logged on node $name"
+ "check that slot $slot invalidation has been logged on node $node_name"
);
+
+ # Check that the invalidation reason is 'inactive_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
}
done_testing();
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
@ 2024-09-16 10:01 ` Bharath Rupireddy <[email protected]>
2024-09-16 11:24 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-09-17 01:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-07 10:03 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
1 sibling, 4 replies; 98+ messages in thread
From: Bharath Rupireddy @ 2024-09-16 10:01 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Thanks for reviewing.
On Mon, Sep 9, 2024 at 1:11 PM Peter Smith <[email protected]> wrote:
>
> 1.
> + Note that the inactive timeout invalidation mechanism is not
> + applicable for slots on the standby server that are being synced
> + from primary server (i.e., standby slots having
>
> nit - /from primary server/from the primary server/
+1
> 2. ReplicationSlotAcquire
>
> + errmsg("can no longer get changes from replication slot \"%s\"",
> + NameStr(s->data.name)),
> + errdetail("This slot has been invalidated because it was inactive
> for longer than the amount of time specified by \"%s\".",
> + "replication_slot_inactive_timeout.")));
>
> nit - "replication_slot_inactive_timeout." - should be no period
> inside that GUC name literal
Typo. Fixed.
> 3. ReportSlotInvalidation
>
> I didn't understand why there was a hint for:
> "You might need to increase \"%s\".", "max_slot_wal_keep_size"
>
> Why aren't these similar cases consistent?
It looks misleading and not very useful. What happens if the removed
WAL (that's needed for the slot) is put back into pg_wal somehow (by
manually copying from archive or by some tool/script)? Can the slot
invalidated due to wal_removed start sending WAL to its clients?
> But you don't have an equivalent hint for timeout invalidation:
> "You might need to increase \"%s\".", "replication_slot_inactive_timeout"
I removed this per review comments upthread.
> 4. RestoreSlotFromDisk
>
> + /* Use the same inactive_since time for all the slots. */
> + if (now == 0)
> + now = GetCurrentTimestamp();
> +
>
> Is the deferred assignment really necessary? Why not just
> unconditionally assign the 'now' just before the for-loop? Or even at
> the declaration? e.g. The 'replication_slot_inactive_timeout' is
> measured in seconds so I don't think 'inactive_since' being wrong by a
> millisecond here will make any difference.
Moved it before the for-loop.
> 5. ReplicationSlotSetInactiveSince
>
> +/*
> + * Set slot's inactive_since property unless it was previously invalidated due
> + * to inactive timeout.
> + */
> +static inline void
> +ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz *now,
> + bool acquire_lock)
> +{
> + if (acquire_lock)
> + SpinLockAcquire(&s->mutex);
> +
> + if (s->data.invalidated != RS_INVAL_INACTIVE_TIMEOUT)
> + s->inactive_since = *now;
> +
> + if (acquire_lock)
> + SpinLockRelease(&s->mutex);
> +}
>
> Is the logic correct? What if the slot was already invalid due to some
> reason other than RS_INVAL_INACTIVE_TIMEOUT? Is an Assert needed?
Hm. Since invalidated slots can't be acquired and made active, not
modifying inactive_since irrespective of invalidation reason looks
good to me.
Please find the attached v46 patch having changes for the above review
comments and your test review comments and Shveta's review comments.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v46-0001-Introduce-inactive_timeout-based-replication-slo.patch (33.5K, ../../CALj2ACXjkM4nEE5dXeV=NA2NmbE3AmonKEgudYBMqU3qO9-9fw@mail.gmail.com/2-v46-0001-Introduce-inactive_timeout-based-replication-slo.patch)
download | inline diff:
From 360a41be6ba3b8e6ba17b26d7be7b7a63c56b485 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Mon, 16 Sep 2024 09:34:33 +0000
Subject: [PATCH v46] Introduce inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage for instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named replication_slot_inactive_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are inactive for longer than this amount of
time.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila
Reviewed-by: Ajin Cherian, Shveta Malik, Peter Smith
Discussion: https://www.postgresql.org/message-id/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://www.postgresql.org/message-id/CA%2BTgmoZTbaaEjSZUG1FL0mzxAdN3qmXksO3O9_PZhEuXTkVnRQ%40mail.gmail.com
Discussion: https://www.postgresql.org/message-id/202403260841.5jcv7ihniccy%40alvherre.pgsql
---
doc/src/sgml/config.sgml | 35 +++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 17 +-
src/backend/replication/slot.c | 166 +++++++++--
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 25 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 266 ++++++++++++++++++
13 files changed, 505 insertions(+), 35 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 0aec11f443..bce7ecdc86 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4556,6 +4556,41 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the inactive timeout invalidation mechanism. This parameter can only
+ be set in the <filename>postgresql.conf</filename> file or on the
+ server command line.
+ </para>
+
+ <para>
+ Slot invalidation due to inactive timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from the primary server (i.e., standby slots having
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+ value <literal>true</literal>).
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 634a4c0fab..5633429eef 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2618,6 +2618,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for longer than the amount of time specified by the
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index f9649eec1a..d5ea75065b 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -448,7 +448,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -667,7 +667,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
@@ -1510,7 +1510,7 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
static void
update_synced_slots_inactive_since(void)
{
- TimestampTz now = 0;
+ TimestampTz now;
/*
* We need to update inactive_since only when we are promoting standby to
@@ -1525,6 +1525,9 @@ update_synced_slots_inactive_since(void)
/* The slot sync worker or SQL function mustn't be running by now */
Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (int i = 0; i < max_replication_slots; i++)
@@ -1539,13 +1542,7 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- /* Use the same inactive_since time for all the slots. */
- if (now == 0)
- now = GetCurrentTimestamp();
-
- SpinLockAcquire(&s->mutex);
- s->inactive_since = now;
- SpinLockRelease(&s->mutex);
+ ReplicationSlotSetInactiveSince(s, &now, true);
}
}
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 6828100cf1..851120e6d2 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -535,9 +537,12 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +620,22 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated due to inactive timeout.
+ */
+ if (error_if_invalid &&
+ s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive for longer than the amount of time specified by \"%s\".",
+ "replication_slot_inactive_timeout")));
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -703,16 +724,12 @@ ReplicationSlotRelease(void)
*/
SpinLockAcquire(&slot->mutex);
slot->active_pid = 0;
- slot->inactive_since = now;
+ ReplicationSlotSetInactiveSince(slot, &now, false);
SpinLockRelease(&slot->mutex);
ConditionVariableBroadcast(&slot->active_cv);
}
else
- {
- SpinLockAcquire(&slot->mutex);
- slot->inactive_since = now;
- SpinLockRelease(&slot->mutex);
- }
+ ReplicationSlotSetInactiveSince(slot, &now, true);
MyReplicationSlot = NULL;
@@ -785,7 +802,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +829,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, false);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1508,7 +1525,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1538,6 +1556,15 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for longer than the amount of time specified by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1554,6 +1581,31 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
pfree(err_detail.data);
}
+/*
+ * Is this replication slot allowed for inactive timeout invalidation check?
+ *
+ * Inactive timeout invalidation is allowed only when:
+ *
+ * 1. Inactive timeout is set
+ * 2. Slot is inactive
+ * 3. Server is not in recovery
+ * 4. Slot is not being synced from the primary
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s)
+{
+ return (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !RecoveryInProgress() &&
+ !s->data.synced);
+}
+
/*
* Helper for InvalidateObsoleteReplicationSlots
*
@@ -1581,6 +1633,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1588,6 +1641,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1598,6 +1652,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ SlotInactiveTimeoutCheckAllowed(s))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1651,6 +1715,28 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ if (!SlotInactiveTimeoutCheckAllowed(s))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1676,11 +1762,13 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1735,7 +1823,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1781,7 +1870,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1804,6 +1894,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1856,7 +1947,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1914,6 +2006,38 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
@@ -2208,6 +2332,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2368,6 +2493,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be \"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2400,7 +2528,7 @@ RestoreSlotFromDisk(const char *name)
* slot from the disk into memory. Whoever acquires the slot i.e.
* makes the slot active will reset it.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ ReplicationSlotSetInactiveSince(slot, &now, false);
restored = true;
break;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index c7bfbb15e0..b1b7b075bd 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -540,7 +540,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index c5f1009f37..61a0e38715 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -844,7 +844,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1462,7 +1462,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index c54b08fe18..82956d58d3 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -299,7 +299,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 686309db58..5e27cd3270 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 667e0dc40a..deca3a4aeb 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -335,6 +335,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 45582cf9d8..8ffd06dd9f 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -224,6 +226,25 @@ typedef struct ReplicationSlotCtlData
ReplicationSlot replication_slots[1];
} ReplicationSlotCtlData;
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz *now,
+ bool acquire_lock)
+{
+ if (s->data.invalidated != RS_INVAL_NONE)
+ return;
+
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ s->inactive_since = *now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
+
/*
* Pointers to shared memory
*/
@@ -233,6 +254,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -249,7 +271,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index 712924c2fa..c45a5106f4 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -52,6 +52,7 @@ tests += {
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
't/043_wal_replay_wait.pl',
+ 't/050_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..c53b5b3dbf
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,266 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start
+#
+# Test invalidation of streaming standby slot and logical failover slot on the
+# primary due to inactive timeout. Also, test logical failover slot synced to
+# the standby from the primary doesn't get invalidated on its own, but gets the
+# invalidated state from the primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot1'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
+);
+
+# Create standby slot on the primary
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot1', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until the standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+my $logstart = -s $standby1->logfile;
+
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $inactive_timeout = 1;
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced
+ AND NOT temporary
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'logical slot sync_slot1 is synced to standby');
+
+# Give enough time
+sleep($inactive_timeout+1);
+
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby. So, we must not see invalidation message in server
+# log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"sync_slot1\"",
+ $logstart),
+ 'check that synced slot sync_slot1 has not been invalidated on standby'
+);
+
+$logstart = -s $primary->logfile;
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for logical failover slot to become inactive on the primary. Note that
+# nobody has acquired the slot yet, so it must get invalidated due to
+# inactive timeout.
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+ $inactive_timeout);
+
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to inactive timeout. The standby must just
+# sync the invalidated state.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for sync_slot1 invalidation to be synced on standby";
+
+# Make the standby slot on the primary inactive and check for invalidation
+$standby1->stop;
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# =============================================================================
+# Testcase start
+# Invalidate logical subscriber slot due to inactive timeout.
+
+my $publisher = $primary;
+
+# Prepare for test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+is($result, qq(5), "check initial copy was done");
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Make subscriber slot on publisher inactive and check for invalidation
+$subscriber->stop;
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+
+ # Wait for slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND active = 'f' AND
+ inactive_since IS NOT NULL;
+ ])
+ or die
+ "Timed out while waiting for slot $slot to become inactive on node $node_name";
+
+ trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout);
+
+ # Check that an invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
+ ok( $stderr =~
+ /can no longer get changes from replication slot "$slot"/,
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
+}
+
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
+
+ # Give enough time to avoid multiple checkpoints
+ sleep($inactive_timeout + 1);
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot\"",
+ $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot invalidation has been logged on node $node_name"
+ );
+
+ # Check that the invalidation reason is 'inactive_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
+}
+
+done_testing();
--
2.43.0
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-16 11:24 ` Amit Kapila <[email protected]>
2024-09-16 17:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
3 siblings, 1 reply; 98+ messages in thread
From: Amit Kapila @ 2024-09-16 11:24 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
<[email protected]> wrote:
>
> Please find the attached v46 patch having changes for the above review
> comments and your test review comments and Shveta's review comments.
>
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +620,22 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated due to inactive timeout.
+ */
+ if (error_if_invalid &&
+ s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive
for longer than the amount of time specified by \"%s\".",
+ "replication_slot_inactive_timeout")));
+ }
Why raise the ERROR just for timeout invalidation here and why not if
the slot is invalidated for other reasons? This raises the question of
what happens before this patch if the invalid slot is used from places
where we call ReplicationSlotAcquire(). I did a brief code analysis
and found that for StartLogicalReplication(), even if the error won't
occur in ReplicationSlotAcquire(), it would have been caught in
CreateDecodingContext(). I think that is where we should also add this
new error. Similarly, pg_logical_slot_get_changes_guts() and other
logical replication functions should be calling
CreateDecodingContext() which can raise the new ERROR. I am not sure
about how the invalid slots are handled during physical replication,
please check the behavior of that before this patch.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-16 11:24 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
@ 2024-09-16 17:10 ` Bharath Rupireddy <[email protected]>
2024-09-18 12:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-09-16 17:10 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Thanks for looking into this.
On Mon, Sep 16, 2024 at 4:54 PM Amit Kapila <[email protected]> wrote:
>
> Why raise the ERROR just for timeout invalidation here and why not if
> the slot is invalidated for other reasons? This raises the question of
> what happens before this patch if the invalid slot is used from places
> where we call ReplicationSlotAcquire(). I did a brief code analysis
> and found that for StartLogicalReplication(), even if the error won't
> occur in ReplicationSlotAcquire(), it would have been caught in
> CreateDecodingContext(). I think that is where we should also add this
> new error. Similarly, pg_logical_slot_get_changes_guts() and other
> logical replication functions should be calling
> CreateDecodingContext() which can raise the new ERROR. I am not sure
> about how the invalid slots are handled during physical replication,
> please check the behavior of that before this patch.
When physical slots are invalidated due to wal_removed reason, the failure
happens at a much later point for the streaming standbys while reading the
requested WAL files like the following:
2024-09-16 16:29:52.416 UTC [876059] FATAL: could not receive data from
WAL stream: ERROR: requested WAL segment 000000010000000000000005 has
already been removed
2024-09-16 16:29:52.416 UTC [872418] LOG: waiting for WAL to become
available at 0/5002000
At this point, despite the slot being invalidated, its wal_status can still
come back to 'unreserved' even from 'lost', and the standby can catch up if
removed WAL files are copied either by manually or by a tool/script to the
primary's pg_wal directory. IOW, the physical slots invalidated due to
wal_removed are *somehow* recoverable unlike the logical slots.
IIUC, the invalidation of a slot implies that it is not guaranteed to hold
any resources like WAL and XMINs. Does it also imply that the slot must be
unusable?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-16 11:24 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-09-16 17:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-18 12:10 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Amit Kapila @ 2024-09-18 12:10 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Sep 16, 2024 at 10:41 PM Bharath Rupireddy
<[email protected]> wrote:
>
> Thanks for looking into this.
>
> On Mon, Sep 16, 2024 at 4:54 PM Amit Kapila <[email protected]> wrote:
> >
> > Why raise the ERROR just for timeout invalidation here and why not if
> > the slot is invalidated for other reasons? This raises the question of
> > what happens before this patch if the invalid slot is used from places
> > where we call ReplicationSlotAcquire(). I did a brief code analysis
> > and found that for StartLogicalReplication(), even if the error won't
> > occur in ReplicationSlotAcquire(), it would have been caught in
> > CreateDecodingContext(). I think that is where we should also add this
> > new error. Similarly, pg_logical_slot_get_changes_guts() and other
> > logical replication functions should be calling
> > CreateDecodingContext() which can raise the new ERROR. I am not sure
> > about how the invalid slots are handled during physical replication,
> > please check the behavior of that before this patch.
>
> When physical slots are invalidated due to wal_removed reason, the failure happens at a much later point for the streaming standbys while reading the requested WAL files like the following:
>
> 2024-09-16 16:29:52.416 UTC [876059] FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000000000000005 has already been removed
> 2024-09-16 16:29:52.416 UTC [872418] LOG: waiting for WAL to become available at 0/5002000
>
> At this point, despite the slot being invalidated, its wal_status can still come back to 'unreserved' even from 'lost', and the standby can catch up if removed WAL files are copied either by manually or by a tool/script to the primary's pg_wal directory. IOW, the physical slots invalidated due to wal_removed are *somehow* recoverable unlike the logical slots.
>
> IIUC, the invalidation of a slot implies that it is not guaranteed to hold any resources like WAL and XMINs. Does it also imply that the slot must be unusable?
>
If we can't hold the dead rows against xmin of the invalid slot, then
how can we make it usable even after copying the required WAL?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-17 01:27 ` Peter Smith <[email protected]>
3 siblings, 0 replies; 98+ messages in thread
From: Peter Smith @ 2024-09-17 01:27 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Here are a few comments for the patch v46-0001.
======
src/backend/replication/slot.c
1. ReportSlotInvalidation
On Mon, Sep 16, 2024 at 8:01 PM Bharath Rupireddy
<[email protected]> wrote:
>
> On Mon, Sep 9, 2024 at 1:11 PM Peter Smith <[email protected]> wrote:
> > 3. ReportSlotInvalidation
> >
> > I didn't understand why there was a hint for:
> > "You might need to increase \"%s\".", "max_slot_wal_keep_size"
> >
> > Why aren't these similar cases consistent?
>
> It looks misleading and not very useful. What happens if the removed
> WAL (that's needed for the slot) is put back into pg_wal somehow (by
> manually copying from archive or by some tool/script)? Can the slot
> invalidated due to wal_removed start sending WAL to its clients?
>
> > But you don't have an equivalent hint for timeout invalidation:
> > "You might need to increase \"%s\".", "replication_slot_inactive_timeout"
>
> I removed this per review comments upthread.
IIUC the errors are quite similar, so my previous review comment was
mostly about the unexpected inconsistency of why one of them has a
hint and the other one does not. I don't have a strong opinion about
whether they should both *have* or *not have* hints, so long as they
are treated the same.
If you think the current code hint is not useful then maybe we need a
new thread to address that existing issue. For example, maybe it
should be removed or reworded.
~~~
2. InvalidatePossiblyObsoleteSlot:
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ if (!SlotInactiveTimeoutCheckAllowed(s))
+ break;
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
nit - it might be tidier to avoid multiple breaks by just combining
these conditions. See the nitpick attachment.
~~~
3.
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
nit - use comment wording "inactive slot timeout has occurred", to
make it identical to the comment in slot.h
======
src/test/recovery/t/050_invalidate_slots.pl
4.
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby. So, we must not see invalidation message in server
+# log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"sync_slot1\"",
+ $logstart),
+ 'check that synced slot sync_slot1 has not been invalidated on standby'
+);
+
It seems kind of brittle to check the logs for something that is NOT
there because any change to the message will make this accidentally
pass. Apart from that, it might anyway be more efficient just to check
the pg_replication_slots again to make sure the 'invalidation_reason
remains' still NULL.
======
Please see the attachment which implements some of the nit changes
mentioned above.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 851120e..0076e4b 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -1716,15 +1716,12 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
invalidation_cause = cause;
break;
case RS_INVAL_INACTIVE_TIMEOUT:
-
- if (!SlotInactiveTimeoutCheckAllowed(s))
- break;
-
/*
* Check if the slot needs to be invalidated due to
* replication_slot_inactive_timeout GUC.
*/
- if (TimestampDifferenceExceeds(s->inactive_since, now,
+ if (SlotInactiveTimeoutCheckAllowed(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
replication_slot_inactive_timeout * 1000))
{
invalidation_cause = cause;
@@ -1894,7 +1891,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
- * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout has occurred
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
index c53b5b3..e2fdd52 100644
--- a/src/test/recovery/t/050_invalidate_slots.pl
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -87,7 +87,7 @@ is( $standby1->safe_psql(
'logical slot sync_slot1 is synced to standby');
# Give enough time
-sleep($inactive_timeout+1);
+sleep($inactive_timeout + 1);
# Despite inactive timeout being set, the synced slot won't get invalidated on
# its own on the standby. So, we must not see invalidation message in server
Attachments:
[text/plain] PS_NITPICKS_20240917_SLOT_TIMEOUT_v46.txt (1.8K, ../../CAHut+Ps=x+2Hq5ue0YppOeDZqgHTnyw=u+vs-qy0JRjKaeJtew@mail.gmail.com/2-PS_NITPICKS_20240917_SLOT_TIMEOUT_v46.txt)
download | inline diff:
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 851120e..0076e4b 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -1716,15 +1716,12 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
invalidation_cause = cause;
break;
case RS_INVAL_INACTIVE_TIMEOUT:
-
- if (!SlotInactiveTimeoutCheckAllowed(s))
- break;
-
/*
* Check if the slot needs to be invalidated due to
* replication_slot_inactive_timeout GUC.
*/
- if (TimestampDifferenceExceeds(s->inactive_since, now,
+ if (SlotInactiveTimeoutCheckAllowed(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
replication_slot_inactive_timeout * 1000))
{
invalidation_cause = cause;
@@ -1894,7 +1891,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
- * - RS_INVAL_INACTIVE_TIMEOUT: inactive timeout occurs
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout has occurred
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
index c53b5b3..e2fdd52 100644
--- a/src/test/recovery/t/050_invalidate_slots.pl
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -87,7 +87,7 @@ is( $standby1->safe_psql(
'logical slot sync_slot1 is synced to standby');
# Give enough time
-sleep($inactive_timeout+1);
+sleep($inactive_timeout + 1);
# Despite inactive timeout being set, the synced slot won't get invalidated on
# its own on the standby. So, we must not see invalidation message in server
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-18 06:51 ` shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:35 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
3 siblings, 2 replies; 98+ messages in thread
From: shveta malik @ 2024-09-18 06:51 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
<[email protected]> wrote:
>
> Hi,
>
>
> Please find the attached v46 patch having changes for the above review
> comments and your test review comments and Shveta's review comments.
>
Thanks for addressing comments.
Is there a reason that we don't support this invalidation on hot
standby for non-synced slots? Shouldn't we support this time-based
invalidation there too just like other invalidations?
thanks
Shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-18 09:19 ` shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: shveta malik @ 2024-09-18 09:19 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Wed, Sep 18, 2024 at 12:21 PM shveta malik <[email protected]> wrote:
>
> On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
> <[email protected]> wrote:
> >
> > Hi,
> >
> >
> > Please find the attached v46 patch having changes for the above review
> > comments and your test review comments and Shveta's review comments.
> >
>
> Thanks for addressing comments.
>
> Is there a reason that we don't support this invalidation on hot
> standby for non-synced slots? Shouldn't we support this time-based
> invalidation there too just like other invalidations?
>
Now since we are not changing inactive_since once it is invalidated,
we are not even initializing it during restart; and thus later when
someone tries to use slot, it leads to assert in
ReplicationSlotAcquire() ( Assert(s->inactive_since > 0);
Steps:
--Disable logical subscriber and let the slot on publisher gets
invalidated due to inactive_timeout.
--Enable the logical subscriber again.
--Restart publisher.
a) We should initialize inactive_since when
ReplicationSlotSetInactiveSince() is called from RestoreSlotFromDisk()
even though it is invalidated.
b) And shall we mention in the doc of 'active_since', that once the
slot is invalidated, this value will remain unchanged until we
shutdown the server. On server restart, it is initialized to start
time. Thought?
thanks
Shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-18 10:01 ` shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-11 12:42 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
0 siblings, 2 replies; 98+ messages in thread
From: shveta malik @ 2024-09-18 10:01 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Wed, Sep 18, 2024 at 2:49 PM shveta malik <[email protected]> wrote:
>
> > > Please find the attached v46 patch having changes for the above review
> > > comments and your test review comments and Shveta's review comments.
> > >
When the synced slot is marked as 'inactive_timeout' invalidated on
hot standby due to invalidation of publisher 's failover slot, the
former starts showing NULL' inactive_since'. Is this intentional
behaviour? I feel inactive_since should be non-NULL here too?
Thoughts?
physical standby:
postgres=# select slot_name, inactive_since, invalidation_reason,
failover, synced from pg_replication_slots;
slot_name | inactive_since |
invalidation_reason | failover | synced
-------------+----------------------------------+---------------------+----------+--------
sub2 | 2024-09-18 15:20:04.364998+05:30 | | t | t
sub3 | 2024-09-18 15:20:04.364953+05:30 | | t | t
After sync of invalidation_reason:
slot_name | inactive_since | invalidation_reason |
failover | synced
-------------+----------------------------------+---------------------+----------+--------
sub2 | | inactive_timeout | t | t
sub3 | | inactive_timeout | t | t
thanks
shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-19 04:10 ` shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: shveta malik @ 2024-09-19 04:10 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Wed, Sep 18, 2024 at 3:31 PM shveta malik <[email protected]> wrote:
>
> On Wed, Sep 18, 2024 at 2:49 PM shveta malik <[email protected]> wrote:
> >
> > > > Please find the attached v46 patch having changes for the above review
> > > > comments and your test review comments and Shveta's review comments.
> > > >
>
When we promote hot standby with synced logical slots to become new
primary, the logical slots are never invalidated with
'inactive_timeout' on new primary. It seems the check in
SlotInactiveTimeoutCheckAllowed() is wrong. We should allow
invalidation of slots on primary even if they are marked as 'synced'.
Please see [4].
I have raised 4 issues so far on v46, the first 3 are in [1],[2],[3].
Once all these are addressed, I can continue reviewing further.
[1]: https://www.postgresql.org/message-id/CAJpy0uAwxc49Dz6t%3D-y_-z-MU%2BA4RWX4BR3Zri_jj2qgGMq_8g%40mail...
[2]: https://www.postgresql.org/message-id/CAJpy0uC6nN3SLbEuCvz7-CpaPdNdXxH%3DfeW5MhYQch-JWV0tLg%40mail.g...
[3]: https://www.postgresql.org/message-id/CAJpy0uBXXJC6f04%2BFU1axKaU%2Bp78wN0SEhUNE9XoqbjXj%3Dhhgw%40ma...
[4]:
--------------------
postgres=# select pg_is_in_recovery();
--------
f
postgres=# show replication_slot_inactive_timeout;
replication_slot_inactive_timeout
-----------------------------------
10s
postgres=# select slot_name, inactive_since, invalidation_reason,
synced from pg_replication_slots;
slot_name | inactive_since | invalidation_reason | synced
-------------+----------------------------------+---------------------+----------+--------
mysubnew1_1 | 2024-09-19 09:04:09.714283+05:30 | | t
postgres=# select now();
now
----------------------------------
2024-09-19 09:06:28.871354+05:30
postgres=# checkpoint;
CHECKPOINT
postgres=# select slot_name, inactive_since, invalidation_reason,
synced from pg_replication_slots;
slot_name | inactive_since | invalidation_reason | synced
-------------+----------------------------------+---------------------+----------+--------
mysubnew1_1 | 2024-09-19 09:04:09.714283+05:30 | | t
--------------------
thanks
Shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-11-13 09:30 ` Nisha Moond <[email protected]>
2024-11-13 23:58 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
0 siblings, 2 replies; 98+ messages in thread
From: Nisha Moond @ 2024-11-13 09:30 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Please find the v48 patch attached.
On Thu, Sep 19, 2024 at 9:40 AM shveta malik <[email protected]> wrote:
>
> When we promote hot standby with synced logical slots to become new
> primary, the logical slots are never invalidated with
> 'inactive_timeout' on new primary. It seems the check in
> SlotInactiveTimeoutCheckAllowed() is wrong. We should allow
> invalidation of slots on primary even if they are marked as 'synced'.
fixed.
> I have raised 4 issues so far on v46, the first 3 are in [1],[2],[3].
> Once all these are addressed, I can continue reviewing further.
>
Fixed issues reported in [1], [2].
[1]: https://www.postgresql.org/message-id/CAJpy0uAwxc49Dz6t%3D-y_-z-MU%2BA4RWX4BR3Zri_jj2qgGMq_8g%40mail...
[2]: https://www.postgresql.org/message-id/CAJpy0uC6nN3SLbEuCvz7-CpaPdNdXxH%3DfeW5MhYQch-JWV0tLg%40mail.g...
Attachments:
[application/octet-stream] v48-0001-Introduce-inactive_timeout-based-replication-slo.patch (33.8K, ../../CABdArM44oW0dQa6Vv5c4xcY54rvhaKsBOpbDVaYzf0F3hUZFPw@mail.gmail.com/2-v48-0001-Introduce-inactive_timeout-based-replication-slo.patch)
download | inline diff:
From 5c933e5231bb20e9eb3b0089665de76adb896cca Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Thu, 7 Nov 2024 12:09:47 +0530
Subject: [PATCH v48] Introduce inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage for instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named replication_slot_inactive_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are inactive for longer than this amount of
time.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.
---
doc/src/sgml/config.sgml | 35 +++
doc/src/sgml/system-views.sgml | 12 +-
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 17 +-
src/backend/replication/slot.c | 162 +++++++++--
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 25 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 267 ++++++++++++++++++
13 files changed, 505 insertions(+), 37 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a84e60c09b..25ca232a02 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4585,6 +4585,41 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the inactive timeout invalidation mechanism. This parameter can only
+ be set in the <filename>postgresql.conf</filename> file or on the
+ server command line.
+ </para>
+
+ <para>
+ Slot invalidation due to inactive timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from the primary server (i.e., standby slots having
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+ value <literal>true</literal>).
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 61d28e701f..d16496a941 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2567,8 +2567,9 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</para>
<para>
The time since the slot has become inactive.
- <literal>NULL</literal> if the slot is currently being used.
- Note that for slots on the standby that are being synced from a
+ <literal>NULL</literal> if the slot is currently being used. Once the
+ slot is invalidated, this value will remain unchanged until we shutdown
+ the server. Note that for slots on the standby that are being synced from a
primary server (whose <structfield>synced</structfield> field is
<literal>true</literal>), the
<structfield>inactive_since</structfield> indicates the last
@@ -2618,6 +2619,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for longer than the amount of time specified by the
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index d62186a510..4443bd53b4 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -446,7 +446,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -665,7 +665,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
static void
update_synced_slots_inactive_since(void)
{
- TimestampTz now = 0;
+ TimestampTz now;
/*
* We need to update inactive_since only when we are promoting standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
/* The slot sync worker or SQL function mustn't be running by now */
Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (int i = 0; i < max_replication_slots; i++)
@@ -1537,13 +1540,7 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- /* Use the same inactive_since time for all the slots. */
- if (now == 0)
- now = GetCurrentTimestamp();
-
- SpinLockAcquire(&s->mutex);
- s->inactive_since = now;
- SpinLockRelease(&s->mutex);
+ ReplicationSlotSetInactiveSince(s, &now, true);
}
}
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 6828100cf1..df76291b7d 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -535,9 +537,12 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +620,22 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated due to inactive timeout.
+ */
+ if (error_if_invalid &&
+ s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive for longer than the amount of time specified by \"%s\".",
+ "replication_slot_inactive_timeout")));
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -703,16 +724,12 @@ ReplicationSlotRelease(void)
*/
SpinLockAcquire(&slot->mutex);
slot->active_pid = 0;
- slot->inactive_since = now;
+ ReplicationSlotSetInactiveSince(slot, &now, false);
SpinLockRelease(&slot->mutex);
ConditionVariableBroadcast(&slot->active_cv);
}
else
- {
- SpinLockAcquire(&slot->mutex);
- slot->inactive_since = now;
- SpinLockRelease(&slot->mutex);
- }
+ ReplicationSlotSetInactiveSince(slot, &now, true);
MyReplicationSlot = NULL;
@@ -785,7 +802,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +829,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, false);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1508,7 +1525,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1538,6 +1556,15 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for longer than the amount of time specified by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1554,6 +1581,29 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
pfree(err_detail.data);
}
+/*
+ * Is this replication slot allowed for inactive timeout invalidation check?
+ *
+ * Inactive timeout invalidation is allowed only when:
+ *
+ * 1. Inactive timeout is set
+ * 2. Slot is inactive
+ * 3. Server is in recovery and slot is not being synced from the primary
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s)
+{
+ return (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced));
+}
+
/*
* Helper for InvalidateObsoleteReplicationSlots
*
@@ -1581,6 +1631,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1588,6 +1639,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1598,6 +1650,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ SlotInactiveTimeoutCheckAllowed(s))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1651,6 +1713,26 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (SlotInactiveTimeoutCheckAllowed(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1676,11 +1758,13 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1735,7 +1819,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1781,7 +1866,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1804,6 +1890,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout has occurred
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1856,7 +1943,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1914,6 +2002,38 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
@@ -2208,6 +2328,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2368,6 +2489,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be \"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2400,7 +2524,7 @@ RestoreSlotFromDisk(const char *name)
* slot from the disk into memory. Whoever acquires the slot i.e.
* makes the slot active will reset it.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;
restored = true;
break;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 488a161b3e..578cff64c8 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -536,7 +536,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 371eef3ddd..b36ae90b2c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -816,7 +816,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1434,7 +1434,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 8a45b5827e..3322848e03 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -298,7 +298,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 8a67f01200..367f510118 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 39a3ac2312..7c6ae1baa2 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 45582cf9d8..8ffd06dd9f 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -224,6 +226,25 @@ typedef struct ReplicationSlotCtlData
ReplicationSlot replication_slots[1];
} ReplicationSlotCtlData;
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz *now,
+ bool acquire_lock)
+{
+ if (s->data.invalidated != RS_INVAL_NONE)
+ return;
+
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ s->inactive_since = *now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
+
/*
* Pointers to shared memory
*/
@@ -233,6 +254,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -249,7 +271,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..708a2a3798 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -51,6 +51,7 @@ tests += {
't/040_standby_failover_slots_sync.pl',
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
+ 't/050_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..270f87c10a
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,267 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start
+#
+# Test invalidation of streaming standby slot and logical failover slot on the
+# primary due to inactive timeout. Also, test logical failover slot synced to
+# the standby from the primary doesn't get invalidated on its own, but gets the
+# invalidated state from the primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot1'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
+);
+
+# Create standby slot on the primary
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot1', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until the standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+my $logstart = -s $standby1->logfile;
+
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $inactive_timeout = 1;
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced
+ AND NOT temporary
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'logical slot sync_slot1 is synced to standby');
+
+# Give enough time
+sleep($inactive_timeout + 1);
+
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby. So, we must not see invalidation message in server
+# log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'check that synced slot sync_slot1 has not been invalidated on standby');
+
+$logstart = -s $primary->logfile;
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for logical failover slot to become inactive on the primary. Note that
+# nobody has acquired the slot yet, so it must get invalidated due to
+# inactive timeout.
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+ $inactive_timeout);
+
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to inactive timeout. The standby must just
+# sync the invalidated state.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for sync_slot1 invalidation to be synced on standby";
+
+# Make the standby slot on the primary inactive and check for invalidation
+$standby1->stop;
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# =============================================================================
+# Testcase start
+# Invalidate logical subscriber slot due to inactive timeout.
+
+my $publisher = $primary;
+
+# Prepare for test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+is($result, qq(5), "check initial copy was done");
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Make subscriber slot on publisher inactive and check for invalidation
+$subscriber->stop;
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+
+ # Wait for slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND active = 'f' AND
+ inactive_since IS NOT NULL;
+ ])
+ or die
+ "Timed out while waiting for slot $slot to become inactive on node $node_name";
+
+ trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout);
+
+ # Check that an invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
+ ok( $stderr =~ /can no longer get changes from replication slot "$slot"/,
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
+}
+
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
+
+ # Give enough time to avoid multiple checkpoints
+ sleep($inactive_timeout + 1);
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot\"", $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot invalidation has been logged on node $node_name"
+ );
+
+ # Check that the invalidation reason is 'inactive_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-13 23:58 ` Peter Smith <[email protected]>
2024-11-19 07:17 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: Peter Smith @ 2024-11-13 23:58 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi Nisha.
Thanks for the recent patch updates. Here are my review comments for
the latest patch v48-0001.
======
Commit message
1.
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage for instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
~
What do the words "for instance" mean here? Did it mean "per instance"
or "(for example)" or something else?
======
doc/src/sgml/system-views.sgml
2.
<para>
The time since the slot has become inactive.
- <literal>NULL</literal> if the slot is currently being used.
- Note that for slots on the standby that are being synced from a
+ <literal>NULL</literal> if the slot is currently being used. Once the
+ slot is invalidated, this value will remain unchanged until we shutdown
+ the server. Note that for slots on the standby that are being
synced from a
primary server (whose <structfield>synced</structfield> field is
<literal>true</literal>), the
Is this change related to the new inactivity timeout feature or are
you just clarifying the existing behaviour of the 'active_since'
field.
Note there is already another thread [1] created to patch/clarify this
same field. So if you are just clarifying existing behavior then IMO
it would be better if you can to try and get your desired changes
included there quickly before that other patch gets pushed.
~~~
3.
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for longer than the amount of time specified by the
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
Maybe there is a slightly shorter/simpler way to express this. For example,
BEFORE
inactive_timeout means that the slot has been inactive for longer than
the amount of time specified by the replication_slot_inactive_timeout
parameter.
SUGGESTION
inactive_timeout means that the slot has remained inactive beyond the
duration specified by the replication_slot_inactive_timeout parameter.
======
src/backend/replication/slot.c
4.
+int replication_slot_inactive_timeout = 0;
IMO it would be more informative to give the units in the variable
name (but not in the GUC name). e.g.
'replication_slot_inactive_timeout_secs'.
~~~
ReplicationSlotAcquire:
5.
+ *
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
This function comment makes it seem like "invalidated previously"
might mean *any* kind of invalidation, but later in the body of the
function we find the logic is really only used for inactive timeout.
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated due to inactive timeout.
+ */
So, I think a better name for that parameter might be
'error_if_inactive_timeout'
OTOH, if it really is supposed to erro for *any* kind of invalidation
then there needs to be more ereports.
~~~
6.
+ errdetail("This slot has been invalidated because it was inactive
for longer than the amount of time specified by \"%s\".",
This errdetail message seems quite long. I think it can be shortened
like below and still retain exactly the same meaning:
BEFORE:
This slot has been invalidated because it was inactive for longer than
the amount of time specified by \"%s\".
SUGGESTION:
This slot has been invalidated due to inactivity exceeding the time
limit set by "%s".
~~~
ReportSlotInvalidation:
7.
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for longer than the amount of
time specified by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;
Here also as in the above review comment #6 I think the message can be
shorter and still say the same thing
BEFORE:
_("The slot has been inactive since %s for longer than the amount of
time specified by \"%s\"."),
SUGGESTION:
_("The slot has been inactive since %s, exceeding the time limit set
by \"%s\"."),
~~~
SlotInactiveTimeoutCheckAllowed:
8.
+/*
+ * Is this replication slot allowed for inactive timeout invalidation check?
+ *
+ * Inactive timeout invalidation is allowed only when:
+ *
+ * 1. Inactive timeout is set
+ * 2. Slot is inactive
+ * 3. Server is in recovery and slot is not being synced from the primary
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
8a.
Somehow that first sentence seems strange. Would it be better to write it like:
SUGGESTION
Can this replication slot timeout due to inactivity?
~
8b.
AFAICT that reason 3 ("Server is in recovery and slot is not being
synced from the primary") seems not quite worded right...
Should it say more like:
The slot is not being synced from the primary while the server is in recovery
or maybe like:
The slot is not currently being synced from the primary (e.g. not
'synced' is true when server is in recovery)
~
8c.
Similarly, I think something about that "Note that the inactive
timeout invalidation mechanism is not applicable..." paragraph needs
tweaking because IMO that should also now be saying something about
'RecoveryInProgress'.
~~~
9.
+static inline bool
+SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s)
Maybe the function name should be 'IsSlotInactiveTimeoutPossible' or
something better.
~~~
InvalidatePossiblyObsoleteSlot:
10.
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
Since there are no other blank lines anywhere in this switch, the
introduction of this one in v48 looks out of place to me. IMO it would
be more readable if a blank line followed each/every of the breaks,
but then that is not a necessary change for this patch so...
~~~
11.
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
Given this assertion, does it mean that "(s->active_pid == 0)" should
have been another condition done up-front in the function
'SlotInactiveTimeoutCheckAllowed'?
~~~
12.
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
I wasn't sure how this change belongs to this patch, because the logic
of the previous review comment said for the case of invalidation due
to inactivity that active_id must be 0. e.g. Assert(s->active_pid ==
0);
~~~
RestoreSlotFromDisk:
13.
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;
In v47 this assignment used to call the function
'ReplicationSlotSetInactiveSince'. I recognise there is a very subtle
difference between direct assignment and the function, because the
function will skip assignment if the slot is already invalidated.
Anyway, if you are *deliberately* not wanting to call
ReplicationSlotSetInactiveSince here then I think this assignment
should be commented to explain the reason why not, otherwise someone
in the future might be tempted to think it was just an oversight and
add the call back in that you don't want.
======
src/test/recovery/t/050_invalidate_slots.pl
14.
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby. So, we must not see invalidation message in server
+# log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'check that synced slot sync_slot1 has not been invalidated on standby');
+
But, now, we are confirming this by another way -- not checking the
logs here, so the comment "So, we must not see invalidation message in
server log." is no longer appropriate here.
======
[1] https://www.postgresql.org/message-id/flat/CAA4eK1JQFdssaBBh-oQskpKM-UpG8jPyUdtmGWa_0qCDy%2BK7_A%40m...
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-13 23:58 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
@ 2024-11-19 07:17 ` Nisha Moond <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Nisha Moond @ 2024-11-19 07:17 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Nov 14, 2024 at 5:29 AM Peter Smith <[email protected]> wrote:
>
> Hi Nisha.
>
> Thanks for the recent patch updates. Here are my review comments for
> the latest patch v48-0001.
>
Thank you for the review. Comments are addressed in v49 version.
Below is my response to comments that may require further discussion.
> ======
> doc/src/sgml/system-views.sgml
>
> 2.
> <para>
> The time since the slot has become inactive.
> - <literal>NULL</literal> if the slot is currently being used.
> - Note that for slots on the standby that are being synced from a
> + <literal>NULL</literal> if the slot is currently being used. Once the
> + slot is invalidated, this value will remain unchanged until we shutdown
> + the server. Note that for slots on the standby that are being
> synced from a
> primary server (whose <structfield>synced</structfield> field is
> <literal>true</literal>), the
>
> Is this change related to the new inactivity timeout feature or are
> you just clarifying the existing behaviour of the 'active_since'
> field.
>
Yes, this patch introduces inactive_timeout invalidation and prevents
updates to inactive_since for invalid slots. Only a node restart can
modify it, so, I believe we should retain these lines in this patch.
> Note there is already another thread [1] created to patch/clarify this
> same field. So if you are just clarifying existing behavior then IMO
> it would be better if you can to try and get your desired changes
> included there quickly before that other patch gets pushed.
>
Thanks for the reference, I have posted my suggestion on the thread.
>
> ReplicationSlotAcquire:
>
> 5.
> + *
> + * An error is raised if error_if_invalid is true and the slot has been
> + * invalidated previously.
> */
> void
> -ReplicationSlotAcquire(const char *name, bool nowait)
> +ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
>
> This function comment makes it seem like "invalidated previously"
> might mean *any* kind of invalidation, but later in the body of the
> function we find the logic is really only used for inactive timeout.
>
> + /*
> + * An error is raised if error_if_invalid is true and the slot has been
> + * previously invalidated due to inactive timeout.
> + */
>
> So, I think a better name for that parameter might be
> 'error_if_inactive_timeout'
>
> OTOH, if it really is supposed to erro for *any* kind of invalidation
> then there needs to be more ereports.
>
+1 to the idea.
I have created a separate patch v49-0001 adding more ereports for all
kinds of invalidations.
> ~~~
> SlotInactiveTimeoutCheckAllowed:
>
> 8.
> +/*
> + * Is this replication slot allowed for inactive timeout invalidation check?
> + *
> + * Inactive timeout invalidation is allowed only when:
> + *
> + * 1. Inactive timeout is set
> + * 2. Slot is inactive
> + * 3. Server is in recovery and slot is not being synced from the primary
> + *
> + * Note that the inactive timeout invalidation mechanism is not
> + * applicable for slots on the standby server that are being synced
> + * from the primary server (i.e., standby slots having 'synced' field 'true').
> + * Synced slots are always considered to be inactive because they don't
> + * perform logical decoding to produce changes.
> + */
>
> 8a.
> Somehow that first sentence seems strange. Would it be better to write it like:
>
> SUGGESTION
> Can this replication slot timeout due to inactivity?
>
I feel the suggestion is not very clear on the purpose of the
function, This function doesn't check inactivity or decide slot
timeout invalidation. It only pre-checks if the slot qualifies for an
inactivity check, which the caller will perform.
As I have changed function name too as per commnet#9, I used the following -
"Is inactive timeout invalidation possible for this replication slot?"
Thoughts?
> ~
> 8c.
> Similarly, I think something about that "Note that the inactive
> timeout invalidation mechanism is not applicable..." paragraph needs
> tweaking because IMO that should also now be saying something about
> 'RecoveryInProgress'.
>
'RecoveryInProgress' check indicates that the server is a standby, and
the mentioned paragraph uses the term "standby" to describe the
condition. It seems unnecessary to mention RecoveryInProgress
separately.
> ~~~
>
> InvalidatePossiblyObsoleteSlot:
>
> 10.
> break;
> + case RS_INVAL_INACTIVE_TIMEOUT:
> +
> + /*
> + * Check if the slot needs to be invalidated due to
> + * replication_slot_inactive_timeout GUC.
> + */
>
> Since there are no other blank lines anywhere in this switch, the
> introduction of this one in v48 looks out of place to me.
pgindent automatically added this blank line after 'case
RS_INVAL_INACTIVE_TIMEOUT'.
> IMO it would
> be more readable if a blank line followed each/every of the breaks,
> but then that is not a necessary change for this patch so...
>
Since it's not directly related to the patch, I feel it might be best
to leave it as is for now.
> ~~~
>
> 11.
> + /*
> + * Invalidation due to inactive timeout implies that
> + * no one is using the slot.
> + */
> + Assert(s->active_pid == 0);
>
> Given this assertion, does it mean that "(s->active_pid == 0)" should
> have been another condition done up-front in the function
> 'SlotInactiveTimeoutCheckAllowed'?
>
I don't think it's a good idea to check (s->active_pid == 0) upfront,
before the timeout-invalidation check. AFAIU, this assertion is meant
to ensure active_pid = 0 only if the slot is going to be invalidated,
i.e., when the following condition is true:
TimestampDifferenceExceeds(s->inactive_since, now,
replication_slot_inactive_timeout_sec * 1000)
Thoughts? Open to others' opinions too.
> ~~~
>
> 12.
> /*
> - * If the slot can be acquired, do so and mark it invalidated
> - * immediately. Otherwise we'll signal the owning process, below, and
> - * retry.
> + * If the slot can be acquired, do so and mark it as invalidated. If
> + * the slot is already ours, mark it as invalidated. Otherwise, we'll
> + * signal the owning process below and retry.
> */
> - if (active_pid == 0)
> + if (active_pid == 0 ||
> + (MyReplicationSlot == s &&
> + active_pid == MyProcPid))
>
> I wasn't sure how this change belongs to this patch, because the logic
> of the previous review comment said for the case of invalidation due
> to inactivity that active_id must be 0. e.g. Assert(s->active_pid ==
> 0);
>
I don't fully understand the purpose of this change yet. I'll look
into it further and get back.
> ~~~
>
> RestoreSlotFromDisk:
>
> 13.
> - slot->inactive_since = GetCurrentTimestamp();
> + slot->inactive_since = now;
>
> In v47 this assignment used to call the function
> 'ReplicationSlotSetInactiveSince'. I recognise there is a very subtle
> difference between direct assignment and the function, because the
> function will skip assignment if the slot is already invalidated.
> Anyway, if you are *deliberately* not wanting to call
> ReplicationSlotSetInactiveSince here then I think this assignment
> should be commented to explain the reason why not, otherwise someone
> in the future might be tempted to think it was just an oversight and
> add the call back in that you don't want.
>
Added comment saying avoid using ReplicationSlotSetInactiveSince()
here as it will skip the invalid slots.
~~~~
--
Thanks,
Nisha
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-13 23:58 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-11-19 07:17 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-29 12:36 ` Nisha Moond <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Nisha Moond @ 2024-11-29 12:36 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Nov 19, 2024 at 12:47 PM Nisha Moond <[email protected]> wrote:
>
> On Thu, Nov 14, 2024 at 5:29 AM Peter Smith <[email protected]> wrote:
> >
> >
> > 12.
> > /*
> > - * If the slot can be acquired, do so and mark it invalidated
> > - * immediately. Otherwise we'll signal the owning process, below, and
> > - * retry.
> > + * If the slot can be acquired, do so and mark it as invalidated. If
> > + * the slot is already ours, mark it as invalidated. Otherwise, we'll
> > + * signal the owning process below and retry.
> > */
> > - if (active_pid == 0)
> > + if (active_pid == 0 ||
> > + (MyReplicationSlot == s &&
> > + active_pid == MyProcPid))
> >
> > I wasn't sure how this change belongs to this patch, because the logic
> > of the previous review comment said for the case of invalidation due
> > to inactivity that active_id must be 0. e.g. Assert(s->active_pid ==
> > 0);
> >
>
> I don't fully understand the purpose of this change yet. I'll look
> into it further and get back.
>
This change applies to all types of invalidation, not just
inactive_timeout case, so moved the change to patch-001. It’s a
general optimization for the case when the current process is the
active PID for the slot.
Also, the Assert(s->active_pid == 0); has been removed (in v50) as it
was unnecessary.
--
Thanks,
Nisha
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-14 03:44 ` vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-19 07:20 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
1 sibling, 2 replies; 98+ messages in thread
From: vignesh C @ 2024-11-14 03:44 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, 13 Nov 2024 at 15:00, Nisha Moond <[email protected]> wrote:
>
> Please find the v48 patch attached.
>
> On Thu, Sep 19, 2024 at 9:40 AM shveta malik <[email protected]> wrote:
> >
> > When we promote hot standby with synced logical slots to become new
> > primary, the logical slots are never invalidated with
> > 'inactive_timeout' on new primary. It seems the check in
> > SlotInactiveTimeoutCheckAllowed() is wrong. We should allow
> > invalidation of slots on primary even if they are marked as 'synced'.
>
> fixed.
>
> > I have raised 4 issues so far on v46, the first 3 are in [1],[2],[3].
> > Once all these are addressed, I can continue reviewing further.
> >
>
> Fixed issues reported in [1], [2].
Few comments:
1) Since we don't change the value of now in
ReplicationSlotSetInactiveSince, the function parameter can be passed
by value:
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz *now,
+ bool
acquire_lock)
+{
+ if (s->data.invalidated != RS_INVAL_NONE)
+ return;
+
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ s->inactive_since = *now;
2) Currently it allows a minimum value of less than 1 second like in
milliseconds, I feel we can have some minimum value at least something
like checkpoint_timeout:
diff --git a/src/backend/utils/misc/guc_tables.c
b/src/backend/utils/misc/guc_tables.c
index 8a67f01200..367f510118 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP,
REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a
replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
3) Since SlotInactiveTimeoutCheckAllowed check is just done above and
the current time has been retrieved can we used "now" variable instead
of SlotInactiveTimeoutCheckAllowed again second time:
@@ -1651,6 +1713,26 @@
InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Check if the slot needs to
be invalidated due to
+ *
replication_slot_inactive_timeout GUC.
+ */
+ if
(SlotInactiveTimeoutCheckAllowed(s) &&
+
TimestampDifferenceExceeds(s->inactive_since, now,
+
replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since =
s->inactive_since;
4) I'm not sure if this change required by this patch or is it a
general optimization, if it is required for this patch we can detail
the comments:
@@ -2208,6 +2328,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2368,6 +2489,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be
\"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2400,7 +2524,7 @@ RestoreSlotFromDisk(const char *name)
* slot from the disk into memory. Whoever acquires
the slot i.e.
* makes the slot active will reset it.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;
5) Why should the slot invalidation be updated during shutdown,
shouldn't the inactive_since value be intact during shutdown?
- <literal>NULL</literal> if the slot is currently being used.
- Note that for slots on the standby that are being synced from a
+ <literal>NULL</literal> if the slot is currently being used. Once the
+ slot is invalidated, this value will remain unchanged until we shutdown
+ the server. Note that for slots on the standby that are being
synced from a
6) New Style of ereport does not need braces around errcode, it can be
changed similarly:
+ if (error_if_invalid &&
+ s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes
from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been
invalidated because it was inactive for longer than the amount of time
specified by \"%s\".",
+
"replication_slot_inactive_timeout")));
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
@ 2024-11-19 07:12 ` Nisha Moond <[email protected]>
2024-11-19 10:23 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
1 sibling, 2 replies; 98+ messages in thread
From: Nisha Moond @ 2024-11-19 07:12 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Attached is the v49 patch set:
- Fixed the bug reported in [1].
- Addressed comments in [2] and [3].
I've split the patch into two, implementing the suggested idea in
comment #5 of [2] separately in 001:
Patch-001: Adds additional error reports (for all invalidation types)
in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
true.
Patch-002: The original patch with comments addressed.
~~~~
[1] https://www.postgresql.org/message-id/CALDaNm2mwkVFLfe8pLcU1W5Oy1vRr1Wzp53XGV08kr4Z2%3DSJpA%40mail.g...
[2] https://www.postgresql.org/message-id/CAHut%2BPt6s-qNPdxH5%3D-fr2QKLEv0h16sQ8EvLiGJ-SdQNS6pbw%40mail...
[3] https://www.postgresql.org/message-id/CALDaNm2VQW_gpOJ-QWkEA_h18DN31ELEz2_7QmwWCAg9%3DZew4A%40mail.g...
--
Thanks,
Nisha
Attachments:
[application/octet-stream] v49-0001-Add-error-handling-while-acquiring-a-replication.patch (7.9K, ../../CABdArM5141FHGvfNYD=pCybLDpmEWbRXFJW3HnEoieMScZ62-g@mail.gmail.com/2-v49-0001-Add-error-handling-while-acquiring-a-replication.patch)
download | inline diff:
From 795a0c837329ec376f1214c4278ca95a2378e8f3 Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Mon, 18 Nov 2024 16:13:26 +0530
Subject: [PATCH v49 1/2] Add error handling while acquiring a replication slot
In ReplicationSlotAcquire(), raise an error for invalid slots if caller
specify error_if_invalid=true.
---
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 4 +-
src/backend/replication/slot.c | 48 +++++++++++++++++--
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/include/replication/slot.h | 3 +-
src/test/recovery/t/019_replslot_limit.pl | 2 +-
8 files changed, 55 insertions(+), 12 deletions(-)
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index d62186a510..69a32422bf 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -446,7 +446,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -665,7 +665,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 6828100cf1..0cbfd4fbaf 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -535,9 +535,12 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +618,45 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated.
+ */
+ if (error_if_invalid &&
+ s->data.invalidated != RS_INVAL_NONE)
+ {
+ StringInfoData err_detail;
+
+ initStringInfo(&err_detail);
+ appendStringInfo(&err_detail, _("This slot has been invalidated because "));
+
+ switch (s->data.invalidated)
+ {
+ case RS_INVAL_WAL_REMOVED:
+ appendStringInfo(&err_detail, _("the required WAL has been removed."));
+ break;
+
+ case RS_INVAL_HORIZON:
+ appendStringInfo(&err_detail, _("the required rows have been removed."));
+ break;
+
+ case RS_INVAL_WAL_LEVEL:
+ appendStringInfo(&err_detail, _("wal_level is insufficient for slot."));
+ break;
+
+ case RS_INVAL_NONE:
+ pg_unreachable();
+ }
+
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail_internal("%s", err_detail.data));
+
+ pfree(err_detail.data);
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +827,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +854,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, false);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 488a161b3e..578cff64c8 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -536,7 +536,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 371eef3ddd..b36ae90b2c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -816,7 +816,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1434,7 +1434,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 8a45b5827e..3322848e03 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -298,7 +298,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 45582cf9d8..2c325fd942 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -249,7 +249,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index efb4ba3af1..333e040e7f 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -234,7 +234,7 @@ my $failed = 0;
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
if ($node_standby->log_contains(
- "requested WAL segment [0-9A-F]+ has already been removed",
+ "This slot has been invalidated because the required WAL has been removed",
$logstart))
{
$failed = 1;
--
2.34.1
[application/octet-stream] v49-0002-Introduce-inactive_timeout-based-replication-slo.patch (29.9K, ../../CABdArM5141FHGvfNYD=pCybLDpmEWbRXFJW3HnEoieMScZ62-g@mail.gmail.com/3-v49-0002-Introduce-inactive_timeout-based-replication-slo.patch)
download | inline diff:
From 2b8637c7654676c8c935506e986344a0b1c36133 Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Mon, 18 Nov 2024 16:54:50 +0530
Subject: [PATCH v49 2/2] Introduce inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage per instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named replication_slot_inactive_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are inactive for longer than this amount of
time.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.
---
doc/src/sgml/config.sgml | 35 +++
doc/src/sgml/system-views.sgml | 12 +-
src/backend/replication/logical/slotsync.c | 13 +-
src/backend/replication/slot.c | 166 +++++++++--
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 22 ++
src/include/utils/guc_hooks.h | 2 +
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 266 ++++++++++++++++++
10 files changed, 503 insertions(+), 27 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a84e60c09b..25ca232a02 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4585,6 +4585,41 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the inactive timeout invalidation mechanism. This parameter can only
+ be set in the <filename>postgresql.conf</filename> file or on the
+ server command line.
+ </para>
+
+ <para>
+ Slot invalidation due to inactive timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from the primary server (i.e., standby slots having
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+ value <literal>true</literal>).
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 61d28e701f..7eec92eac9 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2567,8 +2567,9 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</para>
<para>
The time since the slot has become inactive.
- <literal>NULL</literal> if the slot is currently being used.
- Note that for slots on the standby that are being synced from a
+ <literal>NULL</literal> if the slot is currently being used. Once the
+ slot is invalidated, this value will remain unchanged until we shutdown
+ the server. Note that for slots on the standby that are being synced from a
primary server (whose <structfield>synced</structfield> field is
<literal>true</literal>), the
<structfield>inactive_since</structfield> indicates the last
@@ -2618,6 +2619,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has remained
+ inactive beyond the duration specified by the
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 69a32422bf..4dc2811f10 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
static void
update_synced_slots_inactive_since(void)
{
- TimestampTz now = 0;
+ TimestampTz now;
/*
* We need to update inactive_since only when we are promoting standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
/* The slot sync worker or SQL function mustn't be running by now */
Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (int i = 0; i < max_replication_slots; i++)
@@ -1537,13 +1540,7 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- /* Use the same inactive_since time for all the slots. */
- if (now == 0)
- now = GetCurrentTimestamp();
-
- SpinLockAcquire(&s->mutex);
- s->inactive_since = now;
- SpinLockRelease(&s->mutex);
+ ReplicationSlotSetInactiveSince(s, now, true);
}
}
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 0cbfd4fbaf..6dd5d113a0 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout_sec = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -644,6 +646,11 @@ retry:
appendStringInfo(&err_detail, _("wal_level is insufficient for slot."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ appendStringInfo(&err_detail, _("inactivity exceeded the time limit set by \"%s\"."),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -745,16 +752,12 @@ ReplicationSlotRelease(void)
*/
SpinLockAcquire(&slot->mutex);
slot->active_pid = 0;
- slot->inactive_since = now;
+ ReplicationSlotSetInactiveSince(slot, now, false);
SpinLockRelease(&slot->mutex);
ConditionVariableBroadcast(&slot->active_cv);
}
else
- {
- SpinLockAcquire(&slot->mutex);
- slot->inactive_since = now;
- SpinLockRelease(&slot->mutex);
- }
+ ReplicationSlotSetInactiveSince(slot, now, true);
MyReplicationSlot = NULL;
@@ -1550,7 +1553,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1580,6 +1584,15 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s, exceeding the time limit set by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1596,6 +1609,30 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
pfree(err_detail.data);
}
+/*
+ * Is inactive timeout invalidation possible for this replication slot?
+ *
+ * Inactive timeout invalidation is allowed only when:
+ *
+ * 1. Inactive timeout is set
+ * 2. Slot is inactive
+ * 3. The slot is not being synced from the primary while the server
+ * is in recovery
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+IsSlotInactiveTimeoutPossible(ReplicationSlot *s)
+{
+ return (replication_slot_inactive_timeout_sec > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced));
+}
+
/*
* Helper for InvalidateObsoleteReplicationSlots
*
@@ -1623,6 +1660,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1630,6 +1668,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1640,6 +1679,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ IsSlotInactiveTimeoutPossible(s))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1693,6 +1742,26 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (now &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout_sec * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1718,11 +1787,13 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1777,7 +1848,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1823,7 +1895,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1846,6 +1919,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout has occurred
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1898,7 +1972,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1956,6 +2031,38 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
@@ -2250,6 +2357,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2410,6 +2518,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be \"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2440,9 +2551,11 @@ RestoreSlotFromDisk(const char *name)
/*
* Set the time since the slot has become inactive after loading the
* slot from the disk into memory. Whoever acquires the slot i.e.
- * makes the slot active will reset it.
+ * makes the slot active will reset it. Avoid calling
+ * ReplicationSlotSetInactiveSince() here, as it will not set the time
+ * for invalid slots.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;
restored = true;
break;
@@ -2845,3 +2958,22 @@ WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
ConditionVariableCancelSleep();
}
+
+/*
+ * GUC check_hook for replication_slot_inactive_timeout
+ *
+ * We don't allow the value of replication_slot_inactive_timeout other than 0
+ * during the binary upgrade.
+ */
+bool
+check_replication_slot_inactive_timeout(int *newval, void **extra, GucSource source)
+{
+ if (IsBinaryUpgrade && *newval != 0)
+ {
+ GUC_check_errdetail("\"%s\" must be set to 0 during binary upgrade mode.",
+ "replication_slot_inactive_timeout");
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 8a67f01200..f5a6bf4c61 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout_sec,
+ 0, 0, INT_MAX,
+ check_replication_slot_inactive_timeout, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 39a3ac2312..7c6ae1baa2 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 2c325fd942..98f858659c 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -224,6 +226,25 @@ typedef struct ReplicationSlotCtlData
ReplicationSlot replication_slots[1];
} ReplicationSlotCtlData;
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz now,
+ bool acquire_lock)
+{
+ if (s->data.invalidated != RS_INVAL_NONE)
+ return;
+
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ s->inactive_since = now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
+
/*
* Pointers to shared memory
*/
@@ -233,6 +254,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout_sec;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 5813dba0a2..e36b6cfe21 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,7 @@ extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_replication_slot_inactive_timeout(int *newval, void **extra,
+ GucSource source);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..708a2a3798 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -51,6 +51,7 @@ tests += {
't/040_standby_failover_slots_sync.pl',
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
+ 't/050_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..e4f015e302
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,266 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start
+#
+# Test invalidation of streaming standby slot and logical failover slot on the
+# primary due to inactive timeout. Also, test logical failover slot synced to
+# the standby from the primary doesn't get invalidated on its own, but gets the
+# invalidated state from the primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot1'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
+);
+
+# Create standby slot on the primary
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot1', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until the standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+my $logstart = -s $standby1->logfile;
+
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $inactive_timeout = 1;
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced
+ AND NOT temporary
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'logical slot sync_slot1 is synced to standby');
+
+# Give enough time
+sleep($inactive_timeout + 1);
+
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'check that synced slot sync_slot1 has not been invalidated on standby');
+
+$logstart = -s $primary->logfile;
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for logical failover slot to become inactive on the primary. Note that
+# nobody has acquired the slot yet, so it must get invalidated due to
+# inactive timeout.
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+ $inactive_timeout);
+
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to inactive timeout. The standby must just
+# sync the invalidated state.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for sync_slot1 invalidation to be synced on standby";
+
+# Make the standby slot on the primary inactive and check for invalidation
+$standby1->stop;
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# =============================================================================
+# Testcase start
+# Invalidate logical subscriber slot due to inactive timeout.
+
+my $publisher = $primary;
+
+# Prepare for test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+is($result, qq(5), "check initial copy was done");
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Make subscriber slot on publisher inactive and check for invalidation
+$subscriber->stop;
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+
+ # Wait for slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND active = 'f' AND
+ inactive_since IS NOT NULL;
+ ])
+ or die
+ "Timed out while waiting for slot $slot to become inactive on node $node_name";
+
+ trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout);
+
+ # Check that an invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
+ ok( $stderr =~ /can no longer get changes from replication slot "$slot"/,
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
+}
+
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
+
+ # Give enough time to avoid multiple checkpoints
+ sleep($inactive_timeout + 1);
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot\"", $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot invalidation has been logged on node $node_name"
+ );
+
+ # Check that the invalidation reason is 'inactive_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-19 10:23 ` vignesh C <[email protected]>
1 sibling, 0 replies; 98+ messages in thread
From: vignesh C @ 2024-11-19 10:23 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, 19 Nov 2024 at 12:43, Nisha Moond <[email protected]> wrote:
>
> Attached is the v49 patch set:
> - Fixed the bug reported in [1].
> - Addressed comments in [2] and [3].
>
> I've split the patch into two, implementing the suggested idea in
> comment #5 of [2] separately in 001:
>
> Patch-001: Adds additional error reports (for all invalidation types)
> in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
> true.
> Patch-002: The original patch with comments addressed.
Few comments:
1) I felt this check in wait_for_slot_invalidation is not required as
there is a call to trigger_slot_invalidation which sleeps for
inactive_timeout seconds and ensures checkpoint is triggered, also the
test passes without this:
+ # Wait for slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND active = 'f' AND
+ inactive_since IS NOT NULL;
+ ])
+ or die
+ "Timed out while waiting for slot $slot to become inactive
on node $node_name";
2) Instead of calling this in a loop, won't it be enough to call
checkpoint only once explicitly:
+ for (my $i = 0; $i < 10 *
$PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication
slot \"$slot\"", $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot invalidation has been logged on
node $node_name"
+ );
3) Since pg_sync_replication_slots is a sync call, we can directly use
"is( $standby1->safe_psql('postgres', SELECT COUNT(slot_name) = 1 FROM
pg_replication_slots..." instead of poll_query_until:
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for sync_slot1 invalidation to be synced
on standby";
4) Since this variable is being referred to at many places, how about
changing it to inactive_timeout_1s so that it is easier while
reviewing across many places:
# Set timeout GUC on the standby to verify that the next checkpoint will not
# invalidate synced slots.
my $inactive_timeout = 1;
5) Since we have already tested invalidation of logical replication
slot 'sync_slot1' above, this test might not be required:
+# =============================================================================
+# Testcase start
+# Invalidate logical subscriber slot due to inactive timeout.
+
+my $publisher = $primary;
+
+# Prepare for test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-20 07:59 ` vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: vignesh C @ 2024-11-20 07:59 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, 19 Nov 2024 at 12:43, Nisha Moond <[email protected]> wrote:
>
> Attached is the v49 patch set:
> - Fixed the bug reported in [1].
> - Addressed comments in [2] and [3].
>
> I've split the patch into two, implementing the suggested idea in
> comment #5 of [2] separately in 001:
>
> Patch-001: Adds additional error reports (for all invalidation types)
> in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
> true.
> Patch-002: The original patch with comments addressed.
This Assert can fail:
+ /*
+ * Check if the slot needs to
be invalidated due to
+ *
replication_slot_inactive_timeout GUC.
+ */
+ if (now &&
+
TimestampDifferenceExceeds(s->inactive_since, now,
+
replication_slot_inactive_timeout_sec *
1000))
+ {
+ invalidation_cause = cause;
+ inactive_since =
s->inactive_since;
+
+ /*
+ * Invalidation due to
inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
With the following scenario:
Set replication_slot_inactive_timeout to 10 seconds
-- Create a slot
postgres=# select pg_create_logical_replication_slot ('test',
'pgoutput', true, true);
pg_create_logical_replication_slot
------------------------------------
(test,0/1748068)
(1 row)
-- Wait for 10 seconds and execute checkpoint
postgres=# checkpoint;
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process
exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and
repeat your command.
server closed the connection unexpectedly
The assert fails:
#5 0x00005b074f0c922f in ExceptionalCondition
(conditionName=0x5b074f2f0b4c "s->active_pid == 0",
fileName=0x5b074f2f0010 "slot.c", lineNumber=1762) at assert.c:66
#6 0x00005b074ee26ead in InvalidatePossiblyObsoleteSlot
(cause=RS_INVAL_INACTIVE_TIMEOUT, s=0x740925361780, oldestLSN=0,
dboid=0, snapshotConflictHorizon=0, invalidated=0x7fffaee87e63) at
slot.c:1762
#7 0x00005b074ee273b2 in InvalidateObsoleteReplicationSlots
(cause=RS_INVAL_INACTIVE_TIMEOUT, oldestSegno=0, dboid=0,
snapshotConflictHorizon=0) at slot.c:1952
#8 0x00005b074ee27678 in CheckPointReplicationSlots
(is_shutdown=false) at slot.c:2061
#9 0x00005b074e9dfda7 in CheckPointGuts (checkPointRedo=24412528,
flags=108) at xlog.c:7513
#10 0x00005b074e9df4ad in CreateCheckPoint (flags=108) at xlog.c:7179
#11 0x00005b074edc6bfc in CheckpointerMain (startup_data=0x0,
startup_data_len=0) at checkpointer.c:463
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
@ 2024-11-21 12:05 ` Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-27 03:09 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
0 siblings, 2 replies; 98+ messages in thread
From: Nisha Moond @ 2024-11-21 12:05 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Nov 20, 2024 at 1:29 PM vignesh C <[email protected]> wrote:
>
> On Tue, 19 Nov 2024 at 12:43, Nisha Moond <[email protected]> wrote:
> >
> > Attached is the v49 patch set:
> > - Fixed the bug reported in [1].
> > - Addressed comments in [2] and [3].
> >
> > I've split the patch into two, implementing the suggested idea in
> > comment #5 of [2] separately in 001:
> >
> > Patch-001: Adds additional error reports (for all invalidation types)
> > in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
> > true.
> > Patch-002: The original patch with comments addressed.
>
> This Assert can fail:
>
Attached v50 patch-set addressing review comments in [1] and [2].
Regarding the assert issue reported in [2]:
- For temporary replication slots, the current session's pid serves as
the active_pid for the slot, which is expected behavior.
- Therefore, the ASSERT has been removed in v50. Now, if a temporary
slot qualifies for a timeout invalidation, the holding process will be
terminated, and the slot will be invalidated.
[1] https://www.postgresql.org/message-id/CALDaNm2UUTfJczjR-rEQwKgmx%3DiFnuMnR1cXv7ccB%2BO9P15mYg%40mail...
[2] https://www.postgresql.org/message-id/CALDaNm0g86wD2%3DbQdFOy0smsP0MZWyz0CUqXej%3DQi-hCEeqkag%40mail...
--
Thanks,
Nisha
Attachments:
[application/octet-stream] v50-0001-Add-error-handling-while-acquiring-a-replication.patch (7.9K, ../../CABdArM6A3xsedesUWaQYnOM+mAGGj--N2ohoLrk2tO-W8UfR5g@mail.gmail.com/2-v50-0001-Add-error-handling-while-acquiring-a-replication.patch)
download | inline diff:
From 6df6c33d07ec27a8d633e17c004cf2d6bb61c55b Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Mon, 18 Nov 2024 16:13:26 +0530
Subject: [PATCH v50 1/2] Add error handling while acquiring a replication slot
In ReplicationSlotAcquire(), raise an error for invalid slots if caller
specify error_if_invalid=true.
---
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 4 +-
src/backend/replication/slot.c | 48 +++++++++++++++++--
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/include/replication/slot.h | 3 +-
src/test/recovery/t/019_replslot_limit.pl | 2 +-
8 files changed, 55 insertions(+), 12 deletions(-)
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index d62186a510..69a32422bf 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -446,7 +446,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -665,7 +665,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 6828100cf1..0cbfd4fbaf 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -535,9 +535,12 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +618,45 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated.
+ */
+ if (error_if_invalid &&
+ s->data.invalidated != RS_INVAL_NONE)
+ {
+ StringInfoData err_detail;
+
+ initStringInfo(&err_detail);
+ appendStringInfo(&err_detail, _("This slot has been invalidated because "));
+
+ switch (s->data.invalidated)
+ {
+ case RS_INVAL_WAL_REMOVED:
+ appendStringInfo(&err_detail, _("the required WAL has been removed."));
+ break;
+
+ case RS_INVAL_HORIZON:
+ appendStringInfo(&err_detail, _("the required rows have been removed."));
+ break;
+
+ case RS_INVAL_WAL_LEVEL:
+ appendStringInfo(&err_detail, _("wal_level is insufficient for slot."));
+ break;
+
+ case RS_INVAL_NONE:
+ pg_unreachable();
+ }
+
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail_internal("%s", err_detail.data));
+
+ pfree(err_detail.data);
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +827,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +854,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, false);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 488a161b3e..578cff64c8 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -536,7 +536,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 371eef3ddd..b36ae90b2c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -816,7 +816,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1434,7 +1434,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 8a45b5827e..3322848e03 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -298,7 +298,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 45582cf9d8..2c325fd942 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -249,7 +249,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index efb4ba3af1..333e040e7f 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -234,7 +234,7 @@ my $failed = 0;
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
if ($node_standby->log_contains(
- "requested WAL segment [0-9A-F]+ has already been removed",
+ "This slot has been invalidated because the required WAL has been removed",
$logstart))
{
$failed = 1;
--
2.34.1
[application/octet-stream] v50-0002-Introduce-inactive_timeout-based-replication-slo.patch (27.3K, ../../CABdArM6A3xsedesUWaQYnOM+mAGGj--N2ohoLrk2tO-W8UfR5g@mail.gmail.com/3-v50-0002-Introduce-inactive_timeout-based-replication-slo.patch)
download | inline diff:
From 8871cb932a047f04ca4a6c38d24dacd77b6fef6c Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Mon, 18 Nov 2024 16:54:50 +0530
Subject: [PATCH v50 2/2] Introduce inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage per instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named replication_slot_inactive_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are inactive for longer than this amount of
time.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.
---
doc/src/sgml/config.sgml | 35 ++++
doc/src/sgml/system-views.sgml | 12 +-
src/backend/replication/logical/slotsync.c | 13 +-
src/backend/replication/slot.c | 160 +++++++++++++--
src/backend/utils/misc/guc_tables.c | 12 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 22 ++
src/include/utils/guc_hooks.h | 2 +
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 190 ++++++++++++++++++
10 files changed, 421 insertions(+), 27 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index a84e60c09b..25ca232a02 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4585,6 +4585,41 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the inactive timeout invalidation mechanism. This parameter can only
+ be set in the <filename>postgresql.conf</filename> file or on the
+ server command line.
+ </para>
+
+ <para>
+ Slot invalidation due to inactive timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from the primary server (i.e., standby slots having
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+ value <literal>true</literal>).
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 61d28e701f..7eec92eac9 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2567,8 +2567,9 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</para>
<para>
The time since the slot has become inactive.
- <literal>NULL</literal> if the slot is currently being used.
- Note that for slots on the standby that are being synced from a
+ <literal>NULL</literal> if the slot is currently being used. Once the
+ slot is invalidated, this value will remain unchanged until we shutdown
+ the server. Note that for slots on the standby that are being synced from a
primary server (whose <structfield>synced</structfield> field is
<literal>true</literal>), the
<structfield>inactive_since</structfield> indicates the last
@@ -2618,6 +2619,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has remained
+ inactive beyond the duration specified by the
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 69a32422bf..4dc2811f10 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
static void
update_synced_slots_inactive_since(void)
{
- TimestampTz now = 0;
+ TimestampTz now;
/*
* We need to update inactive_since only when we are promoting standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
/* The slot sync worker or SQL function mustn't be running by now */
Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (int i = 0; i < max_replication_slots; i++)
@@ -1537,13 +1540,7 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- /* Use the same inactive_since time for all the slots. */
- if (now == 0)
- now = GetCurrentTimestamp();
-
- SpinLockAcquire(&s->mutex);
- s->inactive_since = now;
- SpinLockRelease(&s->mutex);
+ ReplicationSlotSetInactiveSince(s, now, true);
}
}
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 0cbfd4fbaf..d65b8a6376 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout_sec = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -644,6 +646,11 @@ retry:
appendStringInfo(&err_detail, _("wal_level is insufficient for slot."));
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ appendStringInfo(&err_detail, _("inactivity exceeded the time limit set by \"%s\"."),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -745,16 +752,12 @@ ReplicationSlotRelease(void)
*/
SpinLockAcquire(&slot->mutex);
slot->active_pid = 0;
- slot->inactive_since = now;
+ ReplicationSlotSetInactiveSince(slot, now, false);
SpinLockRelease(&slot->mutex);
ConditionVariableBroadcast(&slot->active_cv);
}
else
- {
- SpinLockAcquire(&slot->mutex);
- slot->inactive_since = now;
- SpinLockRelease(&slot->mutex);
- }
+ ReplicationSlotSetInactiveSince(slot, now, true);
MyReplicationSlot = NULL;
@@ -1550,7 +1553,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1580,6 +1584,15 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s, exceeding the time limit set by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1596,6 +1609,30 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
pfree(err_detail.data);
}
+/*
+ * Is inactive timeout invalidation possible for this replication slot?
+ *
+ * Inactive timeout invalidation is allowed only when:
+ *
+ * 1. Inactive timeout is set
+ * 2. Slot is inactive
+ * 3. The slot is not being synced from the primary while the server
+ * is in recovery
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+IsSlotInactiveTimeoutPossible(ReplicationSlot *s)
+{
+ return (replication_slot_inactive_timeout_sec > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced));
+}
+
/*
* Helper for InvalidateObsoleteReplicationSlots
*
@@ -1623,6 +1660,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1630,6 +1668,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1640,6 +1679,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ IsSlotInactiveTimeoutPossible(s))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1693,6 +1742,20 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (now &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout_sec * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1718,11 +1781,13 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1777,7 +1842,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1823,7 +1889,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1846,6 +1913,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout has occurred
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1898,7 +1966,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1956,6 +2025,38 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
@@ -2250,6 +2351,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2410,6 +2512,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be \"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2440,9 +2545,11 @@ RestoreSlotFromDisk(const char *name)
/*
* Set the time since the slot has become inactive after loading the
* slot from the disk into memory. Whoever acquires the slot i.e.
- * makes the slot active will reset it.
+ * makes the slot active will reset it. Avoid calling
+ * ReplicationSlotSetInactiveSince() here, as it will not set the time
+ * for invalid slots.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;
restored = true;
break;
@@ -2845,3 +2952,22 @@ WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
ConditionVariableCancelSleep();
}
+
+/*
+ * GUC check_hook for replication_slot_inactive_timeout
+ *
+ * We don't allow the value of replication_slot_inactive_timeout other than 0
+ * during the binary upgrade.
+ */
+bool
+check_replication_slot_inactive_timeout(int *newval, void **extra, GucSource source)
+{
+ if (IsBinaryUpgrade && *newval != 0)
+ {
+ GUC_check_errdetail("The value of \"%s\" must be set to 0 during binary upgrade mode.",
+ "replication_slot_inactive_timeout");
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 8a67f01200..f5a6bf4c61 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout_sec,
+ 0, 0, INT_MAX,
+ check_replication_slot_inactive_timeout, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 39a3ac2312..7c6ae1baa2 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 2c325fd942..98f858659c 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -224,6 +226,25 @@ typedef struct ReplicationSlotCtlData
ReplicationSlot replication_slots[1];
} ReplicationSlotCtlData;
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz now,
+ bool acquire_lock)
+{
+ if (s->data.invalidated != RS_INVAL_NONE)
+ return;
+
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ s->inactive_since = now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
+
/*
* Pointers to shared memory
*/
@@ -233,6 +254,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout_sec;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 5813dba0a2..e36b6cfe21 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,7 @@ extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_replication_slot_inactive_timeout(int *newval, void **extra,
+ GucSource source);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..708a2a3798 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -51,6 +51,7 @@ tests += {
't/040_standby_failover_slots_sync.pl',
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
+ 't/050_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..6a10cd0a54
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,190 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start
+#
+# Test invalidation of streaming standby slot and logical failover slot on the
+# primary due to inactive timeout. Also, test logical failover slot synced to
+# the standby from the primary doesn't get invalidated on its own, but gets the
+# invalidated state from the primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot1'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
+);
+
+# Create standby slot on the primary
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot1', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until the standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+my $logstart = -s $standby1->logfile;
+
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $inactive_timeout_1s = 1;
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout_1s}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced
+ AND NOT temporary
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'logical slot sync_slot1 is synced to standby');
+
+# Give enough time
+sleep($inactive_timeout_1s + 1);
+
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'check that synced slot sync_slot1 has not been invalidated on standby');
+
+$logstart = -s $primary->logfile;
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout_1s}s';
+]);
+$primary->reload;
+
+# Wait for logical failover slot to become inactive on the primary. Note that
+# nobody has acquired the slot yet, so it must get invalidated due to
+# inactive timeout.
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+ $inactive_timeout_1s);
+
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to inactive timeout. The standby must just
+# sync the invalidated state.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason = 'inactive_timeout';}
+ ),
+ "t",
+ 'check that invalidation of synced slot sync_slot1 is synced on standby');
+
+# Make the standby slot on the primary inactive and check for invalidation
+$standby1->stop;
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+ $inactive_timeout_1s);
+
+# Testcase end
+# =============================================================================
+
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout_1s) = @_;
+ my $node_name = $node->name;
+
+ trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout_1s);
+
+ # Check that an invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
+ ok( $stderr =~ /can no longer get changes from replication slot "$slot"/,
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
+}
+
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout_1s) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
+
+ # Give enough time to avoid multiple checkpoints
+ sleep($inactive_timeout_1s + 1);
+
+ # Run a checkpoint
+ $node->safe_psql('postgres', "CHECKPOINT");
+
+ # The slot's invalidation should be logged
+ $node->wait_for_log(qr/invalidating obsolete replication slot \"$slot\"/,
+ $offset);
+
+ # Check that the invalidation reason is 'inactive_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-22 12:13 ` vignesh C <[email protected]>
2024-11-28 09:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: vignesh C @ 2024-11-22 12:13 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, 21 Nov 2024 at 17:35, Nisha Moond <[email protected]> wrote:
>
> On Wed, Nov 20, 2024 at 1:29 PM vignesh C <[email protected]> wrote:
> >
> > On Tue, 19 Nov 2024 at 12:43, Nisha Moond <[email protected]> wrote:
> > >
> > > Attached is the v49 patch set:
> > > - Fixed the bug reported in [1].
> > > - Addressed comments in [2] and [3].
> > >
> > > I've split the patch into two, implementing the suggested idea in
> > > comment #5 of [2] separately in 001:
> > >
> > > Patch-001: Adds additional error reports (for all invalidation types)
> > > in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
> > > true.
> > > Patch-002: The original patch with comments addressed.
> >
> > This Assert can fail:
> >
>
> Attached v50 patch-set addressing review comments in [1] and [2].
We are setting inactive_since when the replication slot is released.
We are marking the slot as inactive only if it has been released.
However, there's a scenario where the network connection between the
publisher and subscriber may be lost where the replication slot is not
released, but no changes are replicated due to the network problem. In
this case, no updates would occur in the replication slot for a period
exceeding the replication_slot_inactive_timeout.
Should we invalidate these replication slots as well, or is it
intentionally left out?
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
@ 2024-11-28 09:13 ` vignesh C <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: vignesh C @ 2024-11-28 09:13 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, 22 Nov 2024 at 17:43, vignesh C <[email protected]> wrote:
>
> On Thu, 21 Nov 2024 at 17:35, Nisha Moond <[email protected]> wrote:
> >
> > On Wed, Nov 20, 2024 at 1:29 PM vignesh C <[email protected]> wrote:
> > >
> > > On Tue, 19 Nov 2024 at 12:43, Nisha Moond <[email protected]> wrote:
> > > >
> > > > Attached is the v49 patch set:
> > > > - Fixed the bug reported in [1].
> > > > - Addressed comments in [2] and [3].
> > > >
> > > > I've split the patch into two, implementing the suggested idea in
> > > > comment #5 of [2] separately in 001:
> > > >
> > > > Patch-001: Adds additional error reports (for all invalidation types)
> > > > in ReplicationSlotAcquire() for invalid slots when error_if_invalid =
> > > > true.
> > > > Patch-002: The original patch with comments addressed.
> > >
> > > This Assert can fail:
> > >
> >
> > Attached v50 patch-set addressing review comments in [1] and [2].
>
> We are setting inactive_since when the replication slot is released.
> We are marking the slot as inactive only if it has been released.
> However, there's a scenario where the network connection between the
> publisher and subscriber may be lost where the replication slot is not
> released, but no changes are replicated due to the network problem. In
> this case, no updates would occur in the replication slot for a period
> exceeding the replication_slot_inactive_timeout.
> Should we invalidate these replication slots as well, or is it
> intentionally left out?
On further thinking, I felt we can keep the current implementation as
is and simply add a brief comment in the code to address this.
Additionally, we can mention it in the commit message for clarity.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-28 09:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
@ 2024-11-29 12:36 ` Nisha Moond <[email protected]>
2024-12-03 07:39 ` RE: Introduce XID age and inactive timeout based replication slot invalidation Hayato Kuroda (Fujitsu) <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Nisha Moond @ 2024-11-29 12:36 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Nov 28, 2024 at 2:44 PM vignesh C <[email protected]> wrote:
>
> >
> > We are setting inactive_since when the replication slot is released.
> > We are marking the slot as inactive only if it has been released.
> > However, there's a scenario where the network connection between the
> > publisher and subscriber may be lost where the replication slot is not
> > released, but no changes are replicated due to the network problem. In
> > this case, no updates would occur in the replication slot for a period
> > exceeding the replication_slot_inactive_timeout.
> > Should we invalidate these replication slots as well, or is it
> > intentionally left out?
>
> On further thinking, I felt we can keep the current implementation as
> is and simply add a brief comment in the code to address this.
> Additionally, we can mention it in the commit message for clarity.
>
Thank you for the clarification. I’ve included the explanatory comment
in patch-002.
Attached the v52 patch-set addressing above as well as all other
comments till now in [1], [2], [3], and [4].
[1] https://www.postgresql.org/message-id/CAHut%2BPto1Yz9Fqp07LLP9uvx3sRHe5SOUKuFM1sUF9QA5aLfBA%40mail.g...
[2] https://www.postgresql.org/message-id/CAHut%2BPs%3DH6EBO1ssGfykrJfUQQGh76L0eKuU5XkR9GMs96ZT3g%40mail...
[3] https://www.postgresql.org/message-id/TYAPR01MB56927564EEE26E5433198405F5292%40TYAPR01MB5692.jpnprd0...
[4] https://www.postgresql.org/message-id/CALDaNm1F2YrswzM_WM37BYmiZ9Cf60UD_mgtm8HnMHRGA7tx4g%40mail.gma...
--
Thanks,
Nisha
Attachments:
[application/octet-stream] v52-0001-Enhance-replication-slot-error-handling-slot-inv.patch (10.3K, ../../CABdArM45=j=DAFTEZA8p=oqYo-XqRwhaPCdKpciQCJx+jNeTJg@mail.gmail.com/2-v52-0001-Enhance-replication-slot-error-handling-slot-inv.patch)
download | inline diff:
From 65e449aae21e77180935bf372ece0ff991f67388 Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Mon, 18 Nov 2024 16:13:26 +0530
Subject: [PATCH v52 1/2] Enhance replication slot error handling, slot
invalidation, and inactive_since setting logic
In ReplicationSlotAcquire(), raise an error for invalid slots if the
caller specifies error_if_invalid=true.
Add check if slot is already acquired, then mark it invalidate directly.
Ensure same inactive_since time for all slots in update_synced_slots_inactive_since()
and RestoreSlotFromDisk().
---
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 9 ++-
src/backend/replication/slot.c | 62 ++++++++++++++++---
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/include/replication/slot.h | 3 +-
src/test/recovery/t/019_replslot_limit.pl | 2 +-
8 files changed, 68 insertions(+), 18 deletions(-)
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index f4f80b2312..0bbd702d32 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -446,7 +446,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -665,7 +665,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
static void
update_synced_slots_inactive_since(void)
{
- TimestampTz now = 0;
+ TimestampTz now;
/*
* We need to update inactive_since only when we are promoting standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
/* The slot sync worker or SQL function mustn't be running by now */
Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (int i = 0; i < max_replication_slots; i++)
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 887e38d56e..c6b15bf601 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -535,9 +535,12 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if error_if_invalid is true and the slot is found to
+ * be invalid.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +618,44 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot is found to
+ * be invalid.
+ */
+ if (error_if_invalid &&
+ s->data.invalidated != RS_INVAL_NONE)
+ {
+ StringInfoData err_detail;
+
+ initStringInfo(&err_detail);
+
+ switch (s->data.invalidated)
+ {
+ case RS_INVAL_WAL_REMOVED:
+ appendStringInfo(&err_detail, _("This slot has been invalidated because the required WAL has been removed."));
+ break;
+
+ case RS_INVAL_HORIZON:
+ appendStringInfo(&err_detail, _("This slot has been invalidated because the required rows have been removed."));
+ break;
+
+ case RS_INVAL_WAL_LEVEL:
+ /* translator: %s is a GUC variable name */
+ appendStringInfo(&err_detail, _("This slot has been invalidated because \"%s\" is insufficient for slot."),
+ "wal_level");
+ break;
+
+ case RS_INVAL_NONE:
+ pg_unreachable();
+ }
+
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail_internal("%s", err_detail.data));
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +826,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +853,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, false);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1676,11 +1717,12 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s && active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -2208,6 +2250,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2368,6 +2411,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be \"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2400,7 +2446,7 @@ RestoreSlotFromDisk(const char *name)
* slot from the disk into memory. Whoever acquires the slot i.e.
* makes the slot active will reset it.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;
restored = true;
break;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 488a161b3e..578cff64c8 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -536,7 +536,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 371eef3ddd..b36ae90b2c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -816,7 +816,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1434,7 +1434,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 8a45b5827e..e8bc986c07 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -298,7 +298,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, true);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index d2cf786fd5..f5f2d22163 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -253,7 +253,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index efb4ba3af1..333e040e7f 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -234,7 +234,7 @@ my $failed = 0;
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
if ($node_standby->log_contains(
- "requested WAL segment [0-9A-F]+ has already been removed",
+ "This slot has been invalidated because the required WAL has been removed",
$logstart))
{
$failed = 1;
--
2.34.1
[application/octet-stream] v52-0002-Introduce-inactive_timeout-based-replication-slo.patch (28.2K, ../../CABdArM45=j=DAFTEZA8p=oqYo-XqRwhaPCdKpciQCJx+jNeTJg@mail.gmail.com/3-v52-0002-Introduce-inactive_timeout-based-replication-slo.patch)
download | inline diff:
From df97e1a5dc72adff0fd0933a16b1736353d8d288 Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Fri, 29 Nov 2024 12:03:41 +0530
Subject: [PATCH v52 2/2] Introduce inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage per instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named replication_slot_inactive_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are inactive for longer than this amount of
time.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.
---
doc/src/sgml/config.sgml | 35 ++++
doc/src/sgml/logical-replication.sgml | 5 +
doc/src/sgml/system-views.sgml | 10 +-
src/backend/replication/logical/slotsync.c | 8 +-
src/backend/replication/slot.c | 153 ++++++++++++--
src/backend/utils/misc/guc_tables.c | 12 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/bin/pg_basebackup/pg_createsubscriber.c | 4 +
src/bin/pg_upgrade/server.c | 7 +
src/include/replication/slot.h | 22 ++
src/include/utils/guc_hooks.h | 2 +
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 190 ++++++++++++++++++
13 files changed, 430 insertions(+), 20 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 76ab72db96..b8d094447b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4585,6 +4585,41 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the inactive timeout invalidation mechanism. This parameter can only
+ be set in the <filename>postgresql.conf</filename> file or on the
+ server command line.
+ </para>
+
+ <para>
+ Slot invalidation due to inactive timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from the primary server (i.e., standby slots having
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+ value <literal>true</literal>).
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 8290cd1a08..8a92a422ef 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -2163,6 +2163,11 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
plus some reserve for table synchronization.
</para>
+ <para>
+ Logical replication slot is also affected by
+ <link linkend="guc-replication-slot-inactive-timeout"><varname>replication_slot_inactive_timeout</varname></link>.
+ </para>
+
<para>
<link linkend="guc-max-wal-senders"><varname>max_wal_senders</varname></link>
should be set to at least the same as
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index a586156614..9e22064de8 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2566,7 +2566,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</para>
<para>
The time when the slot became inactive. <literal>NULL</literal> if the
- slot is currently being streamed.
+ slot is currently being streamed. If the slot becomes invalidated,
+ this value will remain unchanged until server shutdown.
Note that for slots on the standby that are being synced from a
primary server (whose <structfield>synced</structfield> field is
<literal>true</literal>), the <structfield>inactive_since</structfield>
@@ -2620,6 +2621,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has remained
+ inactive beyond the duration specified by the
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 0bbd702d32..9777c6a9cc 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -1540,13 +1540,7 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- /* Use the same inactive_since time for all the slots. */
- if (now == 0)
- now = GetCurrentTimestamp();
-
- SpinLockAcquire(&s->mutex);
- s->inactive_since = now;
- SpinLockRelease(&s->mutex);
+ ReplicationSlotSetInactiveSince(s, now, true);
}
}
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index c6b15bf601..e99d8037f9 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout_ms = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -645,6 +647,12 @@ retry:
"wal_level");
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ /* translator: %s is a GUC variable name */
+ appendStringInfo(&err_detail, _("This slot has been invalidated because inactivity exceeded the time limit set by \"%s\"."),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -744,16 +752,12 @@ ReplicationSlotRelease(void)
*/
SpinLockAcquire(&slot->mutex);
slot->active_pid = 0;
- slot->inactive_since = now;
+ ReplicationSlotSetInactiveSince(slot, now, false);
SpinLockRelease(&slot->mutex);
ConditionVariableBroadcast(&slot->active_cv);
}
else
- {
- SpinLockAcquire(&slot->mutex);
- slot->inactive_since = now;
- SpinLockRelease(&slot->mutex);
- }
+ ReplicationSlotSetInactiveSince(slot, now, true);
MyReplicationSlot = NULL;
@@ -1549,7 +1553,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1579,6 +1584,16 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ /* translator: second %s is a GUC variable name */
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s, exceeding the time limit set by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1595,6 +1610,30 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
pfree(err_detail.data);
}
+/*
+ * Is inactive timeout invalidation possible for this replication slot?
+ *
+ * Inactive timeout invalidation is allowed only when:
+ *
+ * 1. Inactive timeout is set
+ * 2. Slot is inactive
+ * 3. The slot is not being synced from the primary while the server
+ * is in recovery
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+IsSlotInactiveTimeoutPossible(ReplicationSlot *s)
+{
+ return (replication_slot_inactive_timeout_ms > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced));
+}
+
/*
* Helper for InvalidateObsoleteReplicationSlots
*
@@ -1622,6 +1661,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1629,6 +1669,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1639,6 +1680,15 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1692,6 +1742,21 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(now > 0);
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (IsSlotInactiveTimeoutPossible(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout_ms))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1777,7 +1842,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1823,7 +1889,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1846,6 +1913,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout has occurred
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1898,7 +1966,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1956,6 +2025,45 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ *
+ * XXX: Slot invalidation due to 'inactive_timeout' occurs only for
+ * released slots, based on 'replication_slot_inactive_timeout'.
+ * Active slots in use for replication are excluded, preventing
+ * accidental invalidation. Slots where communication between the
+ * publisher and subscriber is down are also excluded, as they are
+ * managed by the 'wal_sender_timeout'.
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
@@ -2444,7 +2552,9 @@ RestoreSlotFromDisk(const char *name)
/*
* Set the time since the slot has become inactive after loading the
* slot from the disk into memory. Whoever acquires the slot i.e.
- * makes the slot active will reset it.
+ * makes the slot active will reset it. Avoid calling
+ * ReplicationSlotSetInactiveSince() here, as it will not set the time
+ * for invalid slots.
*/
slot->inactive_since = now;
@@ -2849,3 +2959,22 @@ WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
ConditionVariableCancelSleep();
}
+
+/*
+ * GUC check_hook for replication_slot_inactive_timeout
+ *
+ * The replication_slot_inactive_timeout must be disabled (set to 0)
+ * during the binary upgrade.
+ */
+bool
+check_replication_slot_inactive_timeout(int *newval, void **extra, GucSource source)
+{
+ if (IsBinaryUpgrade && *newval != 0)
+ {
+ GUC_check_errdetail("The value of \"%s\" must be set to 0 during binary upgrade mode.",
+ "replication_slot_inactive_timeout");
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9845abd693..40dbf16c04 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3038,6 +3038,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_MS
+ },
+ &replication_slot_inactive_timeout_ms,
+ 0, 0, INT_MAX,
+ check_replication_slot_inactive_timeout, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 407cd1e08c..18596686da 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index e96370a9ec..7a5567ceef 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -1438,6 +1438,10 @@ start_standby_server(const struct CreateSubscriberOptions *opt, bool restricted_
appendPQExpBuffer(pg_ctl_cmd, "\"%s\" start -D ", pg_ctl_path);
appendShellString(pg_ctl_cmd, subscriber_dir);
appendPQExpBuffer(pg_ctl_cmd, " -s -o \"-c sync_replication_slots=off\"");
+
+ /* Prevent unintended slot invalidation */
+ appendPQExpBuffer(pg_ctl_cmd, " -o \"-c replication_slot_inactive_timeout=0\"");
+
if (restricted_access)
{
appendPQExpBuffer(pg_ctl_cmd, " -o \"-p %s\"", opt->sub_port);
diff --git a/src/bin/pg_upgrade/server.c b/src/bin/pg_upgrade/server.c
index 91bcb4dbc7..203ab89706 100644
--- a/src/bin/pg_upgrade/server.c
+++ b/src/bin/pg_upgrade/server.c
@@ -252,6 +252,13 @@ start_postmaster(ClusterInfo *cluster, bool report_and_exit_on_error)
if (GET_MAJOR_VERSION(cluster->major_version) >= 1700)
appendPQExpBufferStr(&pgoptions, " -c max_slot_wal_keep_size=-1");
+ /*
+ * Use replication_slot_inactive_timeout=0 to prevent slot invalidation
+ * due to inactive_timeout by checkpointer process during upgrade.
+ */
+ if (GET_MAJOR_VERSION(cluster->major_version) >= 1800)
+ appendPQExpBufferStr(&pgoptions, " -c replication_slot_inactive_timeout=0");
+
/*
* Use -b to disable autovacuum and logical replication launcher
* (effective in PG17 or later for the latter).
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index f5f2d22163..1a79671bb4 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -228,6 +230,25 @@ typedef struct ReplicationSlotCtlData
ReplicationSlot replication_slots[1];
} ReplicationSlotCtlData;
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz now,
+ bool acquire_lock)
+{
+ if (s->data.invalidated != RS_INVAL_NONE)
+ return;
+
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ s->inactive_since = now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
+
/*
* Pointers to shared memory
*/
@@ -237,6 +258,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout_ms;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 5813dba0a2..e36b6cfe21 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,7 @@ extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_replication_slot_inactive_timeout(int *newval, void **extra,
+ GucSource source);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..708a2a3798 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -51,6 +51,7 @@ tests += {
't/040_standby_failover_slots_sync.pl',
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
+ 't/050_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..861509f050
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,190 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start
+#
+# Test invalidation of streaming standby slot and logical failover slot on the
+# primary due to inactive timeout. Also, test logical failover slot synced to
+# the standby from the primary doesn't get invalidated on its own, but gets the
+# invalidated state from the primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot1'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
+);
+
+# Create standby slot on the primary
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot1', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until the standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+my $logstart = -s $standby1->logfile;
+
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $inactive_timeout_1s = 1;
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout_1s}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced
+ AND NOT temporary
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'logical slot sync_slot1 is synced to standby');
+
+# Give enough time for inactive_since to exceed the timeout
+sleep($inactive_timeout_1s + 1);
+
+# On standby, synced slots are not invalidated by the inactive timeout
+# until the invalidation state is propagated from the primary.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'check that synced slot sync_slot1 has not been invalidated on standby');
+
+$logstart = -s $primary->logfile;
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout_1s}s';
+]);
+$primary->reload;
+
+# Wait for logical failover slot to become inactive on the primary. Note that
+# nobody has acquired the slot yet, so it must get invalidated due to
+# inactive timeout.
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+ $inactive_timeout_1s);
+
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to inactive timeout. The standby must just
+# sync the invalidated state.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason = 'inactive_timeout';}
+ ),
+ "t",
+ 'check that invalidation of synced slot sync_slot1 is synced on standby');
+
+# Make the standby slot on the primary inactive and check for invalidation
+$standby1->stop;
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+ $inactive_timeout_1s);
+
+# Testcase end
+# =============================================================================
+
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+
+ trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout);
+
+ # Check that an invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
+ ok( $stderr =~ /can no longer get changes from replication slot "$slot"/,
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
+}
+
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
+
+ # Give enough time for inactive_since to exceed the timeout
+ sleep($inactive_timeout + 1);
+
+ # Run a checkpoint
+ $node->safe_psql('postgres', "CHECKPOINT");
+
+ # The slot's invalidation should be logged
+ $node->wait_for_log(qr/invalidating obsolete replication slot \"$slot\"/,
+ $offset);
+
+ # Check that the invalidation reason is 'inactive_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* RE: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-28 09:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-12-03 07:39 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-12-04 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-12-03 07:39 UTC (permalink / raw)
To: 'Nisha Moond' <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; vignesh C <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Dear Nisha,
Thanks for updating the patch!
> Fixed. It is reasonable to align with other timeout parameters by
> using milliseconds as the unit.
It looks you just replaced to GUC_UNIT_MS, but the documentation and
postgresql.conf.sample has not been changed yet. They should follow codes.
Anyway, here are other comments, mostly cosmetic.
01. slot.c
```
+int replication_slot_inactive_timeout_ms = 0;
```
According to other lines, we should add a short comment for the GUC.
02. 050_invalidate_slots.pl
Do you have a reason why you use the number 050? I feel it can be 043.
03. 050_invalidate_slots.pl
Also, not sure the file name is correct. This file contains only a slot invalidation due to the
replication_slot_inactive_timeout. But I feel current name is too general.
04. 050_invalidate_slots.pl
```
+use Time::HiRes qw(usleep);
```
This line is not needed because usleep() is not used in this file.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-28 09:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-03 07:39 ` RE: Introduce XID age and inactive timeout based replication slot invalidation Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-12-04 09:30 ` Nisha Moond <[email protected]>
2024-12-04 10:26 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Nisha Moond @ 2024-12-04 09:30 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; vignesh C <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Dec 3, 2024 at 1:09 PM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> Dear Nisha,
>
> Thanks for updating the patch!
>
> > Fixed. It is reasonable to align with other timeout parameters by
> > using milliseconds as the unit.
>
> It looks you just replaced to GUC_UNIT_MS, but the documentation and
> postgresql.conf.sample has not been changed yet. They should follow codes.
> Anyway, here are other comments, mostly cosmetic.
>
Here is v53 patch-set addressing all the comments in [1] and [2].
[1] https://www.postgresql.org/message-id/CAHut%2BPsQM79f34LLBGq4UeRuZ1URWP6JNZtdN2khYPrLc1YqrQ%40mail.g...
[2] https://www.postgresql.org/message-id/TYAPR01MB5692B7687EE7981AA91BA5B9F5362%40TYAPR01MB5692.jpnprd0...
--
Thanks,
Nisha
Attachments:
[application/x-patch] v53-0001-Enhance-replication-slot-error-handling-slot-inv.patch (10.6K, ../../CABdArM4N60XShbmxXgz4z8ENQGPOMhXSKzvmskdv8V=mQsYOOA@mail.gmail.com/2-v53-0001-Enhance-replication-slot-error-handling-slot-inv.patch)
download | inline diff:
From b7e55950ae7577dfc8ae8893157d6b6124fdba01 Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Mon, 18 Nov 2024 16:13:26 +0530
Subject: [PATCH v53 1/2] Enhance replication slot error handling, slot
invalidation, and inactive_since setting logic
In ReplicationSlotAcquire(), raise an error for invalid slots if the
caller specifies error_if_invalid=true.
Add check if the slot is already acquired, then mark it invalidated directly.
Ensure same inactive_since time for all slots in update_synced_slots_inactive_since()
and RestoreSlotFromDisk().
---
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 13 ++--
src/backend/replication/slot.c | 62 ++++++++++++++++---
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/include/replication/slot.h | 3 +-
src/test/recovery/t/019_replslot_limit.pl | 2 +-
8 files changed, 68 insertions(+), 22 deletions(-)
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index f4f80b2312..e3645aea53 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -446,7 +446,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -665,7 +665,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
static void
update_synced_slots_inactive_since(void)
{
- TimestampTz now = 0;
+ TimestampTz now;
/*
* We need to update inactive_since only when we are promoting standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
/* The slot sync worker or SQL function mustn't be running by now */
Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (int i = 0; i < max_replication_slots; i++)
@@ -1537,10 +1540,6 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- /* Use the same inactive_since time for all the slots. */
- if (now == 0)
- now = GetCurrentTimestamp();
-
SpinLockAcquire(&s->mutex);
s->inactive_since = now;
SpinLockRelease(&s->mutex);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 4a206f9527..7e84e46fef 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -535,9 +535,12 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if error_if_invalid is true and the slot is found to
+ * be invalid.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +618,44 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot is found to
+ * be invalid.
+ */
+ if (error_if_invalid &&
+ s->data.invalidated != RS_INVAL_NONE)
+ {
+ StringInfoData err_detail;
+
+ initStringInfo(&err_detail);
+
+ switch (s->data.invalidated)
+ {
+ case RS_INVAL_WAL_REMOVED:
+ appendStringInfo(&err_detail, _("This slot has been invalidated because the required WAL has been removed."));
+ break;
+
+ case RS_INVAL_HORIZON:
+ appendStringInfo(&err_detail, _("This slot has been invalidated because the required rows have been removed."));
+ break;
+
+ case RS_INVAL_WAL_LEVEL:
+ /* translator: %s is a GUC variable name */
+ appendStringInfo(&err_detail, _("This slot has been invalidated because \"%s\" is insufficient for slot."),
+ "wal_level");
+ break;
+
+ case RS_INVAL_NONE:
+ pg_unreachable();
+ }
+
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail_internal("%s", err_detail.data));
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +826,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +853,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, false);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1676,11 +1717,12 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s && active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -2208,6 +2250,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2368,6 +2411,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be \"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2400,7 +2446,7 @@ RestoreSlotFromDisk(const char *name)
* slot from the disk into memory. Whoever acquires the slot i.e.
* makes the slot active will reset it.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;
restored = true;
break;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 488a161b3e..578cff64c8 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -536,7 +536,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 371eef3ddd..b36ae90b2c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -816,7 +816,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1434,7 +1434,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 8a45b5827e..e8bc986c07 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -298,7 +298,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, true);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index d2cf786fd5..f5f2d22163 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -253,7 +253,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index efb4ba3af1..333e040e7f 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -234,7 +234,7 @@ my $failed = 0;
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
if ($node_standby->log_contains(
- "requested WAL segment [0-9A-F]+ has already been removed",
+ "This slot has been invalidated because the required WAL has been removed",
$logstart))
{
$failed = 1;
--
2.34.1
[application/x-patch] v53-0002-Introduce-inactive_timeout-based-replication-slo.patch (28.2K, ../../CABdArM4N60XShbmxXgz4z8ENQGPOMhXSKzvmskdv8V=mQsYOOA@mail.gmail.com/3-v53-0002-Introduce-inactive_timeout-based-replication-slo.patch)
download | inline diff:
From e4d9d988b6b131b3ff1ce9b3d2b9b625b07bc8a7 Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Wed, 4 Dec 2024 12:51:22 +0530
Subject: [PATCH v53 2/2] Introduce inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage per instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named replication_slot_inactive_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are inactive for longer than this amount of
time.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.
---
doc/src/sgml/config.sgml | 35 ++++
doc/src/sgml/logical-replication.sgml | 5 +
doc/src/sgml/system-views.sgml | 10 +-
src/backend/replication/logical/slotsync.c | 4 +-
src/backend/replication/slot.c | 155 ++++++++++++--
src/backend/utils/misc/guc_tables.c | 12 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/bin/pg_basebackup/pg_createsubscriber.c | 4 +
src/bin/pg_upgrade/server.c | 7 +
src/include/replication/slot.h | 22 ++
src/include/utils/guc_hooks.h | 2 +
src/test/recovery/meson.build | 1 +
.../t/043_invalidate_inactive_slots.pl | 189 ++++++++++++++++++
13 files changed, 431 insertions(+), 16 deletions(-)
create mode 100644 src/test/recovery/t/043_invalidate_inactive_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e0c8325a39..b59275fcad 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4593,6 +4593,41 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
+ it is taken as milliseconds. A value of zero (which is default) disables
+ the inactive timeout invalidation mechanism. This parameter can only
+ be set in the <filename>postgresql.conf</filename> file or on the
+ server command line.
+ </para>
+
+ <para>
+ Slot invalidation due to inactive timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from the primary server (i.e., standby slots having
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+ value <literal>true</literal>).
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 8290cd1a08..8a92a422ef 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -2163,6 +2163,11 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
plus some reserve for table synchronization.
</para>
+ <para>
+ Logical replication slot is also affected by
+ <link linkend="guc-replication-slot-inactive-timeout"><varname>replication_slot_inactive_timeout</varname></link>.
+ </para>
+
<para>
<link linkend="guc-max-wal-senders"><varname>max_wal_senders</varname></link>
should be set to at least the same as
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index a586156614..9e22064de8 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2566,7 +2566,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</para>
<para>
The time when the slot became inactive. <literal>NULL</literal> if the
- slot is currently being streamed.
+ slot is currently being streamed. If the slot becomes invalidated,
+ this value will remain unchanged until server shutdown.
Note that for slots on the standby that are being synced from a
primary server (whose <structfield>synced</structfield> field is
<literal>true</literal>), the <structfield>inactive_since</structfield>
@@ -2620,6 +2621,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has remained
+ inactive beyond the duration specified by the
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index e3645aea53..9777c6a9cc 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -1540,9 +1540,7 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- SpinLockAcquire(&s->mutex);
- s->inactive_since = now;
- SpinLockRelease(&s->mutex);
+ ReplicationSlotSetInactiveSince(s, now, true);
}
}
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 7e84e46fef..4816d28b62 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -141,6 +142,9 @@ ReplicationSlot *MyReplicationSlot = NULL;
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/* Invalidate replication slots inactive beyond this time; '0' disables it */
+int replication_slot_inactive_timeout_ms = 0;
+
/*
* This GUC lists streaming replication standby server slot names that
* logical WAL sender processes will wait for.
@@ -645,6 +649,12 @@ retry:
"wal_level");
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ /* translator: %s is a GUC variable name */
+ appendStringInfo(&err_detail, _("This slot has been invalidated because inactivity exceeded the time limit set by \"%s\"."),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -744,16 +754,12 @@ ReplicationSlotRelease(void)
*/
SpinLockAcquire(&slot->mutex);
slot->active_pid = 0;
- slot->inactive_since = now;
+ ReplicationSlotSetInactiveSince(slot, now, false);
SpinLockRelease(&slot->mutex);
ConditionVariableBroadcast(&slot->active_cv);
}
else
- {
- SpinLockAcquire(&slot->mutex);
- slot->inactive_since = now;
- SpinLockRelease(&slot->mutex);
- }
+ ReplicationSlotSetInactiveSince(slot, now, true);
MyReplicationSlot = NULL;
@@ -1549,7 +1555,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1579,6 +1586,16 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ /* translator: second %s is a GUC variable name */
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s, exceeding the time limit set by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1595,6 +1612,30 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
pfree(err_detail.data);
}
+/*
+ * Is inactive timeout invalidation possible for this replication slot?
+ *
+ * Inactive timeout invalidation is allowed only when:
+ *
+ * 1. Inactive timeout is set
+ * 2. Slot is inactive
+ * 3. The slot is not being synced from the primary while the server
+ * is in recovery
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+IsSlotInactiveTimeoutPossible(ReplicationSlot *s)
+{
+ return (replication_slot_inactive_timeout_ms > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced));
+}
+
/*
* Helper for InvalidateObsoleteReplicationSlots
*
@@ -1622,6 +1663,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1629,6 +1671,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1639,6 +1682,15 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1692,6 +1744,21 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(now > 0);
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (IsSlotInactiveTimeoutPossible(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout_ms))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1777,7 +1844,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1823,7 +1891,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1846,6 +1915,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout has occurred
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1898,7 +1968,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1956,6 +2027,45 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ *
+ * XXX: Slot invalidation due to 'inactive_timeout' occurs only for
+ * released slots, based on 'replication_slot_inactive_timeout'.
+ * Active slots in use for replication are excluded, preventing
+ * accidental invalidation. Slots where communication between the
+ * publisher and subscriber is down are also excluded, as they are
+ * managed by the 'wal_sender_timeout'.
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
@@ -2444,7 +2554,9 @@ RestoreSlotFromDisk(const char *name)
/*
* Set the time since the slot has become inactive after loading the
* slot from the disk into memory. Whoever acquires the slot i.e.
- * makes the slot active will reset it.
+ * makes the slot active will reset it. Avoid calling
+ * ReplicationSlotSetInactiveSince() here, as it will not set the time
+ * for invalid slots.
*/
slot->inactive_since = now;
@@ -2839,3 +2951,22 @@ WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
ConditionVariableCancelSleep();
}
+
+/*
+ * GUC check_hook for replication_slot_inactive_timeout
+ *
+ * The replication_slot_inactive_timeout must be disabled (set to 0)
+ * during the binary upgrade.
+ */
+bool
+check_replication_slot_inactive_timeout(int *newval, void **extra, GucSource source)
+{
+ if (IsBinaryUpgrade && *newval != 0)
+ {
+ GUC_check_errdetail("The value of \"%s\" must be set to 0 during binary upgrade mode.",
+ "replication_slot_inactive_timeout");
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 8cf1afbad2..0e00c07f91 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3047,6 +3047,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_MS
+ },
+ &replication_slot_inactive_timeout_ms,
+ 0, 0, INT_MAX,
+ check_replication_slot_inactive_timeout, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index a2ac7575ca..2f0fa84e6d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -337,6 +337,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in milliseconds; 0 disables
# - Primary Server -
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index e96370a9ec..7a5567ceef 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -1438,6 +1438,10 @@ start_standby_server(const struct CreateSubscriberOptions *opt, bool restricted_
appendPQExpBuffer(pg_ctl_cmd, "\"%s\" start -D ", pg_ctl_path);
appendShellString(pg_ctl_cmd, subscriber_dir);
appendPQExpBuffer(pg_ctl_cmd, " -s -o \"-c sync_replication_slots=off\"");
+
+ /* Prevent unintended slot invalidation */
+ appendPQExpBuffer(pg_ctl_cmd, " -o \"-c replication_slot_inactive_timeout=0\"");
+
if (restricted_access)
{
appendPQExpBuffer(pg_ctl_cmd, " -o \"-p %s\"", opt->sub_port);
diff --git a/src/bin/pg_upgrade/server.c b/src/bin/pg_upgrade/server.c
index 91bcb4dbc7..203ab89706 100644
--- a/src/bin/pg_upgrade/server.c
+++ b/src/bin/pg_upgrade/server.c
@@ -252,6 +252,13 @@ start_postmaster(ClusterInfo *cluster, bool report_and_exit_on_error)
if (GET_MAJOR_VERSION(cluster->major_version) >= 1700)
appendPQExpBufferStr(&pgoptions, " -c max_slot_wal_keep_size=-1");
+ /*
+ * Use replication_slot_inactive_timeout=0 to prevent slot invalidation
+ * due to inactive_timeout by checkpointer process during upgrade.
+ */
+ if (GET_MAJOR_VERSION(cluster->major_version) >= 1800)
+ appendPQExpBufferStr(&pgoptions, " -c replication_slot_inactive_timeout=0");
+
/*
* Use -b to disable autovacuum and logical replication launcher
* (effective in PG17 or later for the latter).
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index f5f2d22163..1a79671bb4 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -228,6 +230,25 @@ typedef struct ReplicationSlotCtlData
ReplicationSlot replication_slots[1];
} ReplicationSlotCtlData;
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz now,
+ bool acquire_lock)
+{
+ if (s->data.invalidated != RS_INVAL_NONE)
+ return;
+
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ s->inactive_since = now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
+
/*
* Pointers to shared memory
*/
@@ -237,6 +258,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout_ms;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 5813dba0a2..e36b6cfe21 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,7 @@ extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_replication_slot_inactive_timeout(int *newval, void **extra,
+ GucSource source);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..708a2a3798 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -51,6 +51,7 @@ tests += {
't/040_standby_failover_slots_sync.pl',
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
+ 't/050_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/043_invalidate_inactive_slots.pl b/src/test/recovery/t/043_invalidate_inactive_slots.pl
new file mode 100644
index 0000000000..6aae73a548
--- /dev/null
+++ b/src/test/recovery/t/043_invalidate_inactive_slots.pl
@@ -0,0 +1,189 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+# =============================================================================
+# Testcase start
+#
+# Test invalidation of streaming standby slot and logical failover slot on the
+# primary due to inactive timeout. Also, test logical failover slot synced to
+# the standby from the primary doesn't get invalidated on its own, but gets the
+# invalidated state from the primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot1'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
+);
+
+# Create standby slot on the primary
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot1', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until the standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+my $logstart = -s $standby1->logfile;
+
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $inactive_timeout_1s = 1;
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout_1s}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced
+ AND NOT temporary
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'logical slot sync_slot1 is synced to standby');
+
+# Give enough time for inactive_since to exceed the timeout
+sleep($inactive_timeout_1s + 1);
+
+# On standby, synced slots are not invalidated by the inactive timeout
+# until the invalidation state is propagated from the primary.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'check that synced slot sync_slot1 has not been invalidated on standby');
+
+$logstart = -s $primary->logfile;
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout_1s}s';
+]);
+$primary->reload;
+
+# Wait for logical failover slot to become inactive on the primary. Note that
+# nobody has acquired the slot yet, so it must get invalidated due to
+# inactive timeout.
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+ $inactive_timeout_1s);
+
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to inactive timeout. The standby must just
+# sync the invalidated state.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason = 'inactive_timeout';}
+ ),
+ "t",
+ 'check that invalidation of synced slot sync_slot1 is synced on standby');
+
+# Make the standby slot on the primary inactive and check for invalidation
+$standby1->stop;
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+ $inactive_timeout_1s);
+
+# Testcase end
+# =============================================================================
+
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+
+ trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout);
+
+ # Check that an invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
+ ok( $stderr =~ /can no longer get changes from replication slot "$slot"/,
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
+}
+
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
+
+ # Give enough time for inactive_since to exceed the timeout
+ sleep($inactive_timeout + 1);
+
+ # Run a checkpoint
+ $node->safe_psql('postgres', "CHECKPOINT");
+
+ # The slot's invalidation should be logged
+ $node->wait_for_log(qr/invalidating obsolete replication slot \"$slot\"/,
+ $offset);
+
+ # Check that the invalidation reason is 'inactive_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-28 09:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-03 07:39 ` RE: Introduce XID age and inactive timeout based replication slot invalidation Hayato Kuroda (Fujitsu) <[email protected]>
2024-12-04 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-12-04 10:26 ` vignesh C <[email protected]>
2024-12-05 01:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: vignesh C @ 2024-12-04 10:26 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: Hayato Kuroda (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, 4 Dec 2024 at 15:01, Nisha Moond <[email protected]> wrote:
>
> On Tue, Dec 3, 2024 at 1:09 PM Hayato Kuroda (Fujitsu)
> <[email protected]> wrote:
> >
> > Dear Nisha,
> >
> > Thanks for updating the patch!
> >
> > > Fixed. It is reasonable to align with other timeout parameters by
> > > using milliseconds as the unit.
> >
> > It looks you just replaced to GUC_UNIT_MS, but the documentation and
> > postgresql.conf.sample has not been changed yet. They should follow codes.
> > Anyway, here are other comments, mostly cosmetic.
> >
>
> Here is v53 patch-set addressing all the comments in [1] and [2].
Currently, replication slots are invalidated based on the
replication_slot_inactive_timeout only during a checkpoint. This means
that if the checkpoint_timeout is set to a higher value than the
replication_slot_inactive_timeout, slot invalidation will occur only
when the checkpoint is triggered. Identifying the invalidation slots
might be slightly delayed in this case. As an alternative, users can
forcefully invalidate inactive slots that have exceeded the
replication_slot_inactive_timeout by forcing a checkpoint. I was
thinking we could suggest this in the documentation.
+ <para>
+ Slot invalidation due to inactive timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
We could accurately invalidate the slots using the checkpointer
process by calculating the invalidation time based on the active_since
timestamp and the replication_slot_inactive_timeout, and then set the
checkpointer's main wait-latch accordingly for triggering the next
checkpoint. Ideally, a different process handling this task would be
better, but there is currently no dedicated daemon capable of
identifying and managing slots across streaming replication, logical
replication, and other slots used by plugins. Additionally,
overloading the checkpointer with this responsibility may not be
ideal. As an alternative, we could document about this delay in
identifying and mention that it could be triggered by forceful manual
checkpoint.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-28 09:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-03 07:39 ` RE: Introduce XID age and inactive timeout based replication slot invalidation Hayato Kuroda (Fujitsu) <[email protected]>
2024-12-04 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-04 10:26 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
@ 2024-12-05 01:13 ` Peter Smith <[email protected]>
2024-12-06 05:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Peter Smith @ 2024-12-05 01:13 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Dec 4, 2024 at 9:27 PM vignesh C <[email protected]> wrote:
>
...
>
> Currently, replication slots are invalidated based on the
> replication_slot_inactive_timeout only during a checkpoint. This means
> that if the checkpoint_timeout is set to a higher value than the
> replication_slot_inactive_timeout, slot invalidation will occur only
> when the checkpoint is triggered. Identifying the invalidation slots
> might be slightly delayed in this case. As an alternative, users can
> forcefully invalidate inactive slots that have exceeded the
> replication_slot_inactive_timeout by forcing a checkpoint. I was
> thinking we could suggest this in the documentation.
>
> + <para>
> + Slot invalidation due to inactive timeout occurs during checkpoint.
> + The duration of slot inactivity is calculated using the slot's
> + <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
> + value.
> + </para>
> +
>
> We could accurately invalidate the slots using the checkpointer
> process by calculating the invalidation time based on the active_since
> timestamp and the replication_slot_inactive_timeout, and then set the
> checkpointer's main wait-latch accordingly for triggering the next
> checkpoint. Ideally, a different process handling this task would be
> better, but there is currently no dedicated daemon capable of
> identifying and managing slots across streaming replication, logical
> replication, and other slots used by plugins. Additionally,
> overloading the checkpointer with this responsibility may not be
> ideal. As an alternative, we could document about this delay in
> identifying and mention that it could be triggered by forceful manual
> checkpoint.
>
Hi Vignesh.
I felt that manipulating the checkpoint timing behind the scenes
without the user's consent might be a bit of an overreach.
But there might still be something else we could do:
1. We can add the documentation note like you suggested ("we could
document about this delay in identifying and mention that it could be
triggered by forceful manual checkpoint").
2. We can also detect such delays in the code. When the invalidation
occurs (e.g. code fragment below) we could check if there was some
excessive lag between the slot becoming idle and it being invalidated.
If the lag is too much (whatever "too much" means) we can log a hint
for the user to increase the checkpoint frequency (or whatever else we
might advise them to do).
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (IsSlotInactiveTimeoutPossible(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout_ms))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
pseudo-code:
if (slot invalidation occurred much later after the
replication_slot_inactive_timeout GUC elapsed)
{
elog(LOG, "This slot was inactive for a period of %s. Slot timeout
invalidation only occurs at a checkpoint so if you want inactive slots
to be invalidated in a more timely manner consider reducing the time
between checkpoints or executing a manual checkpoint.
(replication_slot_inactive_timeout = %s; checkpoint_timeout = %s,
....)"
}
+ }
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-28 09:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-03 07:39 ` RE: Introduce XID age and inactive timeout based replication slot invalidation Hayato Kuroda (Fujitsu) <[email protected]>
2024-12-04 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-04 10:26 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-12-05 01:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
@ 2024-12-06 05:34 ` vignesh C <[email protected]>
2024-12-10 11:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: vignesh C @ 2024-12-06 05:34 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: Nisha Moond <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, 5 Dec 2024 at 06:44, Peter Smith <[email protected]> wrote:
>
> On Wed, Dec 4, 2024 at 9:27 PM vignesh C <[email protected]> wrote:
> >
> ...
> >
> > Currently, replication slots are invalidated based on the
> > replication_slot_inactive_timeout only during a checkpoint. This means
> > that if the checkpoint_timeout is set to a higher value than the
> > replication_slot_inactive_timeout, slot invalidation will occur only
> > when the checkpoint is triggered. Identifying the invalidation slots
> > might be slightly delayed in this case. As an alternative, users can
> > forcefully invalidate inactive slots that have exceeded the
> > replication_slot_inactive_timeout by forcing a checkpoint. I was
> > thinking we could suggest this in the documentation.
> >
> > + <para>
> > + Slot invalidation due to inactive timeout occurs during checkpoint.
> > + The duration of slot inactivity is calculated using the slot's
> > + <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
> > + value.
> > + </para>
> > +
> >
> > We could accurately invalidate the slots using the checkpointer
> > process by calculating the invalidation time based on the active_since
> > timestamp and the replication_slot_inactive_timeout, and then set the
> > checkpointer's main wait-latch accordingly for triggering the next
> > checkpoint. Ideally, a different process handling this task would be
> > better, but there is currently no dedicated daemon capable of
> > identifying and managing slots across streaming replication, logical
> > replication, and other slots used by plugins. Additionally,
> > overloading the checkpointer with this responsibility may not be
> > ideal. As an alternative, we could document about this delay in
> > identifying and mention that it could be triggered by forceful manual
> > checkpoint.
> >
>
> Hi Vignesh.
>
> I felt that manipulating the checkpoint timing behind the scenes
> without the user's consent might be a bit of an overreach.
Agree
> But there might still be something else we could do:
>
> 1. We can add the documentation note like you suggested ("we could
> document about this delay in identifying and mention that it could be
> triggered by forceful manual checkpoint").
Yes, that makes sense
> 2. We can also detect such delays in the code. When the invalidation
> occurs (e.g. code fragment below) we could check if there was some
> excessive lag between the slot becoming idle and it being invalidated.
> If the lag is too much (whatever "too much" means) we can log a hint
> for the user to increase the checkpoint frequency (or whatever else we
> might advise them to do).
>
> + /*
> + * Check if the slot needs to be invalidated due to
> + * replication_slot_inactive_timeout GUC.
> + */
> + if (IsSlotInactiveTimeoutPossible(s) &&
> + TimestampDifferenceExceeds(s->inactive_since, now,
> + replication_slot_inactive_timeout_ms))
> + {
> + invalidation_cause = cause;
> + inactive_since = s->inactive_since;
>
> pseudo-code:
> if (slot invalidation occurred much later after the
> replication_slot_inactive_timeout GUC elapsed)
> {
> elog(LOG, "This slot was inactive for a period of %s. Slot timeout
> invalidation only occurs at a checkpoint so if you want inactive slots
> to be invalidated in a more timely manner consider reducing the time
> between checkpoints or executing a manual checkpoint.
> (replication_slot_inactive_timeout = %s; checkpoint_timeout = %s,
> ....)"
> }
>
> + }
Determining the correct time may be challenging for users, as it
depends on when the active_since value is set, as well as when the
checkpoint_timeout occurs and the subsequent checkpoint is triggered.
Even if the user sets it to an appropriate value, there is still a
possibility of delayed identification due to the timing of when the
slot's active_timeout is being set. Including this information in the
documentation should be sufficient.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-28 09:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-03 07:39 ` RE: Introduce XID age and inactive timeout based replication slot invalidation Hayato Kuroda (Fujitsu) <[email protected]>
2024-12-04 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-04 10:26 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-12-05 01:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-12-06 05:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
@ 2024-12-10 11:51 ` Nisha Moond <[email protected]>
2024-12-11 02:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-12-12 04:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
0 siblings, 2 replies; 98+ messages in thread
From: Nisha Moond @ 2024-12-10 11:51 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Peter Smith <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, Dec 6, 2024 at 11:04 AM vignesh C <[email protected]> wrote:
>
>
> Determining the correct time may be challenging for users, as it
> depends on when the active_since value is set, as well as when the
> checkpoint_timeout occurs and the subsequent checkpoint is triggered.
> Even if the user sets it to an appropriate value, there is still a
> possibility of delayed identification due to the timing of when the
> slot's active_timeout is being set. Including this information in the
> documentation should be sufficient.
>
+1
v54 documents this information as suggested.
Attached the v54 patch-set addressing all the comments till now in
[1], [2] and [3].
[1] https://www.postgresql.org/message-id/CALDaNm0mTWwg0z4v-sorq08S2CdZmL2s%2Brh4nHpWeJaBQ2F%2Bmg%40mail...
[2] https://www.postgresql.org/message-id/CALDaNm1STyk%3DS_EAihWP9SowBkS5dJ32JfEqmG5tTeC2Ct39yg%40mail.g...
[3] https://www.postgresql.org/message-id/CAHut%2BPtHbYNxPvtMfs7jARbsVcFXL1%3DC9SO3Q93NgVDgbKN7LQ%40mail...
--
Thanks,
Nisha
Attachments:
[application/x-patch] v54-0001-Enhance-replication-slot-error-handling-slot-inv.patch (10.6K, ../../CABdArM7CPR0Ag6OgtufATUn+x=Q_h3cdBcc39r+xeKYDks5Mfw@mail.gmail.com/2-v54-0001-Enhance-replication-slot-error-handling-slot-inv.patch)
download | inline diff:
From 713871d8cda02f2b70c63983fc49dede3097f016 Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Mon, 18 Nov 2024 16:13:26 +0530
Subject: [PATCH v54 1/2] Enhance replication slot error handling, slot
invalidation, and inactive_since setting logic
In ReplicationSlotAcquire(), raise an error for invalid slots if the
caller specifies error_if_invalid=true.
Add check if the slot is already acquired, then mark it invalidated directly.
Ensure same inactive_since time for all slots in update_synced_slots_inactive_since()
and RestoreSlotFromDisk().
---
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 13 ++--
src/backend/replication/slot.c | 61 ++++++++++++++++---
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/include/replication/slot.h | 3 +-
src/test/recovery/t/019_replslot_limit.pl | 2 +-
8 files changed, 67 insertions(+), 22 deletions(-)
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index f4f80b2312..e3645aea53 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -446,7 +446,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -665,7 +665,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
static void
update_synced_slots_inactive_since(void)
{
- TimestampTz now = 0;
+ TimestampTz now;
/*
* We need to update inactive_since only when we are promoting standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
/* The slot sync worker or SQL function mustn't be running by now */
Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (int i = 0; i < max_replication_slots; i++)
@@ -1537,10 +1540,6 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- /* Use the same inactive_since time for all the slots. */
- if (now == 0)
- now = GetCurrentTimestamp();
-
SpinLockAcquire(&s->mutex);
s->inactive_since = now;
SpinLockRelease(&s->mutex);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 4a206f9527..db94cec5c3 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -535,9 +535,12 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if error_if_invalid is true and the slot is found to
+ * be invalid.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +618,43 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot is found to
+ * be invalid.
+ */
+ if (error_if_invalid && s->data.invalidated != RS_INVAL_NONE)
+ {
+ StringInfoData err_detail;
+
+ initStringInfo(&err_detail);
+
+ switch (s->data.invalidated)
+ {
+ case RS_INVAL_WAL_REMOVED:
+ appendStringInfo(&err_detail, _("This slot has been invalidated because the required WAL has been removed."));
+ break;
+
+ case RS_INVAL_HORIZON:
+ appendStringInfo(&err_detail, _("This slot has been invalidated because the required rows have been removed."));
+ break;
+
+ case RS_INVAL_WAL_LEVEL:
+ /* translator: %s is a GUC variable name */
+ appendStringInfo(&err_detail, _("This slot has been invalidated because \"%s\" is insufficient for slot."),
+ "wal_level");
+ break;
+
+ case RS_INVAL_NONE:
+ pg_unreachable();
+ }
+
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail_internal("%s", err_detail.data));
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +825,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +852,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, false);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1676,11 +1716,12 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s && active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -2208,6 +2249,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2368,6 +2410,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be \"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2400,7 +2445,7 @@ RestoreSlotFromDisk(const char *name)
* slot from the disk into memory. Whoever acquires the slot i.e.
* makes the slot active will reset it.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;
restored = true;
break;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 488a161b3e..578cff64c8 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -536,7 +536,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 371eef3ddd..b36ae90b2c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -816,7 +816,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1434,7 +1434,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 8a45b5827e..e8bc986c07 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -298,7 +298,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, true);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index d2cf786fd5..f5f2d22163 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -253,7 +253,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index efb4ba3af1..333e040e7f 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -234,7 +234,7 @@ my $failed = 0;
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
if ($node_standby->log_contains(
- "requested WAL segment [0-9A-F]+ has already been removed",
+ "This slot has been invalidated because the required WAL has been removed",
$logstart))
{
$failed = 1;
--
2.34.1
[application/x-patch] v54-0002-Introduce-inactive_timeout-based-replication-slo.patch (28.2K, ../../CABdArM7CPR0Ag6OgtufATUn+x=Q_h3cdBcc39r+xeKYDks5Mfw@mail.gmail.com/3-v54-0002-Introduce-inactive_timeout-based-replication-slo.patch)
download | inline diff:
From 7fc20252d8828083613e7948b9d5a48349af0b26 Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Wed, 4 Dec 2024 12:51:22 +0530
Subject: [PATCH v54 2/2] Introduce inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage per instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named idle_replication_slot_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are idle for longer than this amount of
time.
Note that the idle timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.
---
doc/src/sgml/config.sgml | 39 ++++
doc/src/sgml/logical-replication.sgml | 5 +
doc/src/sgml/system-views.sgml | 10 +-
src/backend/replication/logical/slotsync.c | 4 +-
src/backend/replication/slot.c | 155 +++++++++++++--
src/backend/utils/misc/guc_tables.c | 12 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/bin/pg_basebackup/pg_createsubscriber.c | 4 +
src/bin/pg_upgrade/server.c | 7 +
src/include/replication/slot.h | 22 ++
src/include/utils/guc_hooks.h | 2 +
src/test/recovery/meson.build | 1 +
.../t/043_invalidate_inactive_slots.pl | 188 ++++++++++++++++++
13 files changed, 434 insertions(+), 16 deletions(-)
create mode 100644 src/test/recovery/t/043_invalidate_inactive_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index e0c8325a39..a888c709fd 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4593,6 +4593,45 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-idle-replication-slot-timeout" xreflabel="idle_replication_slot_timeout">
+ <term><varname>idle_replication_slot_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>idle_replication_slot_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots that are idle for longer than this
+ amount of time. If this value is specified without units,
+ it is taken as milliseconds. A value of zero (which is default) disables
+ the idle timeout invalidation mechanism. This parameter can only
+ be set in the <filename>postgresql.conf</filename> file or on the
+ server command line.
+ </para>
+
+ <para>
+ Slot invalidation due to idle timeout occurs during checkpoint.
+ If the <varname>checkpoint_timeout</varname> exceeds
+ <varname>idle_replication_slot_timeout</varname>, the slot
+ invalidation will be delayed until the next checkpoint is triggered.
+ To avoid delays, users can force a checkpoint to promptly invalidate
+ inactive slots. The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
+ <para>
+ Note that the idle timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from the primary server (i.e., standby slots having
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+ value <literal>true</literal>).
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml
index 8290cd1a08..158ec18211 100644
--- a/doc/src/sgml/logical-replication.sgml
+++ b/doc/src/sgml/logical-replication.sgml
@@ -2163,6 +2163,11 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER
plus some reserve for table synchronization.
</para>
+ <para>
+ Logical replication slots are also affected by
+ <link linkend="guc-idle-replication-slot-timeout"><varname>idle_replication_slot_timeout</varname></link>.
+ </para>
+
<para>
<link linkend="guc-max-wal-senders"><varname>max_wal_senders</varname></link>
should be set to at least the same as
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index a586156614..611db4e539 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2566,7 +2566,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</para>
<para>
The time when the slot became inactive. <literal>NULL</literal> if the
- slot is currently being streamed.
+ slot is currently being streamed. If the slot becomes invalidated,
+ this value will remain unchanged until server shutdown.
Note that for slots on the standby that are being synced from a
primary server (whose <structfield>synced</structfield> field is
<literal>true</literal>), the <structfield>inactive_since</structfield>
@@ -2620,6 +2621,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>idle_timeout</literal> means that the slot has remained
+ idle beyond the duration specified by the
+ <xref linkend="guc-idle-replication-slot-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index e3645aea53..9777c6a9cc 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -1540,9 +1540,7 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- SpinLockAcquire(&s->mutex);
- s->inactive_since = now;
- SpinLockRelease(&s->mutex);
+ ReplicationSlotSetInactiveSince(s, now, true);
}
}
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index db94cec5c3..41ef8ecc89 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_IDLE_TIMEOUT] = "idle_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_IDLE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -141,6 +142,9 @@ ReplicationSlot *MyReplicationSlot = NULL;
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+/* Invalidate replication slots idle beyond this time; '0' disables it */
+int idle_replication_slot_timeout_ms = 0;
+
/*
* This GUC lists streaming replication standby server slot names that
* logical WAL sender processes will wait for.
@@ -644,6 +648,12 @@ retry:
"wal_level");
break;
+ case RS_INVAL_IDLE_TIMEOUT:
+ /* translator: %s is a GUC variable name */
+ appendStringInfo(&err_detail, _("This slot has been invalidated because inactivity exceeded the time limit set by \"%s\"."),
+ "idle_replication_slot_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -743,16 +753,12 @@ ReplicationSlotRelease(void)
*/
SpinLockAcquire(&slot->mutex);
slot->active_pid = 0;
- slot->inactive_since = now;
+ ReplicationSlotSetInactiveSince(slot, now, false);
SpinLockRelease(&slot->mutex);
ConditionVariableBroadcast(&slot->active_cv);
}
else
- {
- SpinLockAcquire(&slot->mutex);
- slot->inactive_since = now;
- SpinLockRelease(&slot->mutex);
- }
+ ReplicationSlotSetInactiveSince(slot, now, true);
MyReplicationSlot = NULL;
@@ -1548,7 +1554,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1578,6 +1585,16 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
+ case RS_INVAL_IDLE_TIMEOUT:
+ Assert(inactive_since > 0);
+ /* translator: second %s is a GUC variable name */
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s, exceeding the time limit set by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "idle_replication_slot_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1594,6 +1611,30 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
pfree(err_detail.data);
}
+/*
+ * Is idle timeout invalidation possible for this replication slot?
+ *
+ * Idle timeout invalidation is allowed only when:
+ *
+ * 1. Idle timeout is set
+ * 2. Slot is inactive
+ * 3. The slot is not being synced from the primary while the server
+ * is in recovery
+ *
+ * Note that the idle timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+IsSlotIdleTimeoutPossible(ReplicationSlot *s)
+{
+ return (idle_replication_slot_timeout_ms > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced));
+}
+
/*
* Helper for InvalidateObsoleteReplicationSlots
*
@@ -1621,6 +1662,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1628,6 +1670,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1638,6 +1681,15 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_IDLE_TIMEOUT)
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1691,6 +1743,21 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_IDLE_TIMEOUT:
+ Assert(now > 0);
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * idle_replication_slot_timeout GUC.
+ */
+ if (IsSlotIdleTimeoutPossible(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ idle_replication_slot_timeout_ms))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1776,7 +1843,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1822,7 +1890,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1845,6 +1914,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_IDLE_TIMEOUT: idle slot timeout has occurred
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1897,7 +1967,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1955,6 +2026,45 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do idle_timeout invalidation of
+ * thousands of replication slots here. If it is ever proven that this
+ * assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move idle_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ *
+ * XXX: Slot invalidation due to 'idle_timeout' occurs only for
+ * released slots, based on 'idle_replication_slot_timeout'. Active
+ * slots in use for replication are excluded, preventing accidental
+ * invalidation. Slots where communication between the publisher and
+ * subscriber is down are also excluded, as they are managed by the
+ * 'wal_sender_timeout'.
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_IDLE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
@@ -2443,7 +2553,9 @@ RestoreSlotFromDisk(const char *name)
/*
* Set the time since the slot has become inactive after loading the
* slot from the disk into memory. Whoever acquires the slot i.e.
- * makes the slot active will reset it.
+ * makes the slot active will reset it. Avoid calling
+ * ReplicationSlotSetInactiveSince() here, as it will not set the time
+ * for invalid slots.
*/
slot->inactive_since = now;
@@ -2838,3 +2950,22 @@ WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
ConditionVariableCancelSleep();
}
+
+/*
+ * GUC check_hook for idle_replication_slot_timeout
+ *
+ * The idle_replication_slot_timeout must be disabled (set to 0)
+ * during the binary upgrade.
+ */
+bool
+check_idle_replication_slot_timeout(int *newval, void **extra, GucSource source)
+{
+ if (IsBinaryUpgrade && *newval != 0)
+ {
+ GUC_check_errdetail("The value of \"%s\" must be set to 0 during binary upgrade mode.",
+ "idle_replication_slot_timeout");
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 8cf1afbad2..6a4f15b832 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3047,6 +3047,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"idle_replication_slot_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain idle before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_MS
+ },
+ &idle_replication_slot_timeout_ms,
+ 0, 0, INT_MAX,
+ check_idle_replication_slot_timeout, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index a2ac7575ca..6b5c246e8d 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -337,6 +337,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#idle_replication_slot_timeout = 0 # in milliseconds; 0 disables
# - Primary Server -
diff --git a/src/bin/pg_basebackup/pg_createsubscriber.c b/src/bin/pg_basebackup/pg_createsubscriber.c
index e96370a9ec..a9ac00f44f 100644
--- a/src/bin/pg_basebackup/pg_createsubscriber.c
+++ b/src/bin/pg_basebackup/pg_createsubscriber.c
@@ -1438,6 +1438,10 @@ start_standby_server(const struct CreateSubscriberOptions *opt, bool restricted_
appendPQExpBuffer(pg_ctl_cmd, "\"%s\" start -D ", pg_ctl_path);
appendShellString(pg_ctl_cmd, subscriber_dir);
appendPQExpBuffer(pg_ctl_cmd, " -s -o \"-c sync_replication_slots=off\"");
+
+ /* Prevent unintended slot invalidation */
+ appendPQExpBuffer(pg_ctl_cmd, " -o \"-c idle_replication_slot_timeout=0\"");
+
if (restricted_access)
{
appendPQExpBuffer(pg_ctl_cmd, " -o \"-p %s\"", opt->sub_port);
diff --git a/src/bin/pg_upgrade/server.c b/src/bin/pg_upgrade/server.c
index 91bcb4dbc7..90e2b9f188 100644
--- a/src/bin/pg_upgrade/server.c
+++ b/src/bin/pg_upgrade/server.c
@@ -252,6 +252,13 @@ start_postmaster(ClusterInfo *cluster, bool report_and_exit_on_error)
if (GET_MAJOR_VERSION(cluster->major_version) >= 1700)
appendPQExpBufferStr(&pgoptions, " -c max_slot_wal_keep_size=-1");
+ /*
+ * Use idle_replication_slot_timeout=0 to prevent slot invalidation due to
+ * inactive_timeout by checkpointer process during upgrade.
+ */
+ if (GET_MAJOR_VERSION(cluster->major_version) >= 1800)
+ appendPQExpBufferStr(&pgoptions, " -c idle_replication_slot_timeout=0");
+
/*
* Use -b to disable autovacuum and logical replication launcher
* (effective in PG17 or later for the latter).
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index f5f2d22163..98625f9d13 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* idle slot timeout has occurred */
+ RS_INVAL_IDLE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -228,6 +230,25 @@ typedef struct ReplicationSlotCtlData
ReplicationSlot replication_slots[1];
} ReplicationSlotCtlData;
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz now,
+ bool acquire_lock)
+{
+ if (s->data.invalidated != RS_INVAL_NONE)
+ return;
+
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ s->inactive_since = now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
+
/*
* Pointers to shared memory
*/
@@ -237,6 +258,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int idle_replication_slot_timeout_ms;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 5813dba0a2..d7a7dffab5 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,7 @@ extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_idle_replication_slot_timeout(int *newval, void **extra,
+ GucSource source);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..3b8f45c93e 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -51,6 +51,7 @@ tests += {
't/040_standby_failover_slots_sync.pl',
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
+ 't/043_invalidate_inactive_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/043_invalidate_inactive_slots.pl b/src/test/recovery/t/043_invalidate_inactive_slots.pl
new file mode 100644
index 0000000000..a947bdf2e5
--- /dev/null
+++ b/src/test/recovery/t/043_invalidate_inactive_slots.pl
@@ -0,0 +1,188 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+
+# =============================================================================
+# Testcase start
+#
+# Test invalidation of streaming standby slot and logical failover slot on the
+# primary due to idle timeout. Also, test logical failover slot synced to
+# the standby from the primary doesn't get invalidated on its own, but gets the
+# invalidated state from the primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot1'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
+);
+
+# Create standby slot on the primary
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot1', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until the standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+my $logstart = -s $standby1->logfile;
+
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $idle_timeout_1s = 1;
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET idle_replication_slot_timeout TO '${idle_timeout_1s}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced
+ AND NOT temporary
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'logical slot sync_slot1 is synced to standby');
+
+# Give enough time for inactive_since to exceed the timeout
+sleep($idle_timeout_1s + 1);
+
+# On standby, synced slots are not invalidated by the idle timeout
+# until the invalidation state is propagated from the primary.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'check that synced slot sync_slot1 has not been invalidated on standby');
+
+$logstart = -s $primary->logfile;
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET idle_replication_slot_timeout TO '${idle_timeout_1s}s';
+]);
+$primary->reload;
+
+# Wait for logical failover slot to become inactive on the primary. Note that
+# nobody has acquired the slot yet, so it must get invalidated due to
+# idle timeout.
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+ $idle_timeout_1s);
+
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to idle timeout. The standby must just
+# sync the invalidated state.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason = 'idle_timeout';}
+ ),
+ "t",
+ 'check that invalidation of synced slot sync_slot1 is synced on standby');
+
+# Make the standby slot on the primary inactive and check for invalidation
+$standby1->stop;
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart, $idle_timeout_1s);
+
+# Testcase end
+# =============================================================================
+
+# Wait for slot to first become idle and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $idle_timeout) = @_;
+ my $node_name = $node->name;
+
+ trigger_slot_invalidation($node, $slot, $offset, $idle_timeout);
+
+ # Check that an invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
+ ok( $stderr =~ /can no longer get changes from replication slot "$slot"/,
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
+}
+
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $idle_timeout) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
+
+ # Give enough time for inactive_since to exceed the timeout
+ sleep($idle_timeout + 1);
+
+ # Run a checkpoint
+ $node->safe_psql('postgres', "CHECKPOINT");
+
+ # The slot's invalidation should be logged
+ $node->wait_for_log(qr/invalidating obsolete replication slot \"$slot\"/,
+ $offset);
+
+ # Check that the invalidation reason is 'idle_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'idle_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-28 09:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-03 07:39 ` RE: Introduce XID age and inactive timeout based replication slot invalidation Hayato Kuroda (Fujitsu) <[email protected]>
2024-12-04 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-04 10:26 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-12-05 01:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-12-06 05:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-12-10 11:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-12-11 02:44 ` Peter Smith <[email protected]>
1 sibling, 0 replies; 98+ messages in thread
From: Peter Smith @ 2024-12-11 02:44 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: vignesh C <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi Nisha.
Here are some review comments for patch v54-0002.
(I had also checked patch v54-0001, but have no further review
comments for that one).
======
doc/src/sgml/config.sgml
1.
+ <para>
+ Slot invalidation due to idle timeout occurs during checkpoint.
+ If the <varname>checkpoint_timeout</varname> exceeds
+ <varname>idle_replication_slot_timeout</varname>, the slot
+ invalidation will be delayed until the next checkpoint is triggered.
+ To avoid delays, users can force a checkpoint to promptly invalidate
+ inactive slots. The duration of slot inactivity is calculated
using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
The wording of "If the checkpoint_timeout exceeds
idle_replication_slot_timeout, the slot invalidation will be delayed
until the next checkpoint is triggered." seems slightly misleading,
because AFAIK it is not conditional on the GUC value differences like
that -- i.e. slot invalidation is *always* delayed until the next
checkpoint occurs.
SUGGESTION:
Slot invalidation due to idle timeout occurs during checkpoint.
Because checkpoints happen at checkpoint_timeout intervals, there can
be some lag between when the idle_replication_slot_timeout was
exceeded and when the slot invalidation is triggered at the next
checkpoint. To avoid such lags, users can force...
=======
src/backend/replication/slot.c
2. GENERAL
+/* Invalidate replication slots idle beyond this time; '0' disables it */
+int idle_replication_slot_timeout_ms = 0;
I noticed this patch is using a variety of ways of describing the same thing:
* guc var: Invalidate replication slots idle beyond this time...
* guc_tables: ... the amount of time a replication slot can remain
idle before it will be invalidated.
* docs: means that the slot has remained idle beyond the duration
specified by the idle_replication_slot_timeout parameter
* errmsg: ... slot has been invalidated because inactivity exceeded
the time limit set by ...
* etc..
They are all the same, but they are all worded slightly differently:
* "idle" vs "inactivity" vs ...
* "time" vs "amount of time" vs "duration" vs "time limit" vs ...
There may not be a one-size-fits-all, but still, it might be better to
try to search for all different phrasing and use common wording as
much as possible.
~~~
CheckPointReplicationSlots:
3.
+ * XXX: Slot invalidation due to 'idle_timeout' occurs only for
+ * released slots, based on 'idle_replication_slot_timeout'. Active
+ * slots in use for replication are excluded, preventing accidental
+ * invalidation. Slots where communication between the publisher and
+ * subscriber is down are also excluded, as they are managed by the
+ * 'wal_sender_timeout'.
Maybe a slight rewording like below is better. Maybe not. YMMV.
SUGGESTION:
XXX: Slot invalidation due to 'idle_timeout' applies only to released
slots, and is based on the 'idle_replication_slot_timeout' GUC. Active
slots
currently in use for replication are excluded to prevent accidental
invalidation. Slots...
======
src/bin/pg_upgrade/server.c
4.
+ /*
+ * Use idle_replication_slot_timeout=0 to prevent slot invalidation due to
+ * inactive_timeout by checkpointer process during upgrade.
+ */
+ if (GET_MAJOR_VERSION(cluster->major_version) >= 1800)
+ appendPQExpBufferStr(&pgoptions, " -c idle_replication_slot_timeout=0");
+
/inactive_timeout/idle_timeout/
======
src/test/recovery/t/043_invalidate_inactive_slots.pl
5.
+# Wait for slot to first become idle and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $idle_timeout) = @_;
+ my $node_name = $node->name;
AFAICT this 'idle_timeout' parameter is passed units of "seconds", so
it would be better to call it something like 'idle_timeout_s' to make
the units clear.
~~~
6.
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $idle_timeout) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
Ditto above review comment #5 -- better to call it something like
'idle_timeout_s' to make the units clear.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-22 12:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-28 09:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-03 07:39 ` RE: Introduce XID age and inactive timeout based replication slot invalidation Hayato Kuroda (Fujitsu) <[email protected]>
2024-12-04 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-12-04 10:26 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-12-05 01:13 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-12-06 05:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-12-10 11:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-12-12 04:12 ` vignesh C <[email protected]>
1 sibling, 0 replies; 98+ messages in thread
From: vignesh C @ 2024-12-12 04:12 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: Peter Smith <[email protected]>; Hayato Kuroda (Fujitsu) <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, 10 Dec 2024 at 17:21, Nisha Moond <[email protected]> wrote:
>
> On Fri, Dec 6, 2024 at 11:04 AM vignesh C <[email protected]> wrote:
> >
> >
> > Determining the correct time may be challenging for users, as it
> > depends on when the active_since value is set, as well as when the
> > checkpoint_timeout occurs and the subsequent checkpoint is triggered.
> > Even if the user sets it to an appropriate value, there is still a
> > possibility of delayed identification due to the timing of when the
> > slot's active_timeout is being set. Including this information in the
> > documentation should be sufficient.
> >
>
> +1
> v54 documents this information as suggested.
>
> Attached the v54 patch-set addressing all the comments till now in
> [1], [2] and [3].
Now that we support idle_replication_slot_timeout in milliseconds, we
can set this value from 1s to 1ms or 10millseconds and change sleep to
usleep, this will bring down the test execution time significantly:
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $idle_timeout_1s = 1;
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET idle_replication_slot_timeout TO '${idle_timeout_1s}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced
+ AND NOT temporary
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'logical slot sync_slot1 is synced to standby');
+
+# Give enough time for inactive_since to exceed the timeout
+sleep($idle_timeout_1s + 1);
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-27 03:09 ` Peter Smith <[email protected]>
2024-11-27 10:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: Peter Smith @ 2024-11-27 03:09 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: vignesh C <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi Nisha,
Here are some review comments for the patch v50-0002.
======
src/backend/replication/slot.c
InvalidatePossiblyObsoleteSlot:
1.
+ if (now &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout_sec * 1000))
Previously this was using an additional call to SlotInactiveTimeoutCheckAllowed:
+ if (SlotInactiveTimeoutCheckAllowed(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
Is it OK to skip that call? e.g. can the slot fields possibly change
between assigning the 'now' and acquiring the mutex? If not, then the
current code is fine. The only reason for asking is because it is
slightly suspicious that it was not done this "easy" way in the first
place.
~~~
check_replication_slot_inactive_timeout:
2.
+/*
+ * GUC check_hook for replication_slot_inactive_timeout
+ *
+ * We don't allow the value of replication_slot_inactive_timeout other than 0
+ * during the binary upgrade.
+ */
The "We don't allow..." sentence seems like a backward way of saying:
The value of replication_slot_inactive_timeout must be set to 0 during
the binary upgrade.
======
src/test/recovery/t/050_invalidate_slots.pl
3.
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby.
What does "on its own" mean here? Do you mean it won't get invalidated
unless the invalidation state is propagated from the primary? Maybe
the comment can be clearer.
~
4.
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout_1s) = @_;
+ my $node_name = $node->name;
+
It was OK to change the variable name to 'inactive_timeout_1s' outside
of here, but within the subroutine, I don't think it is appropriate
because this is a parameter that potentially could have any value.
~
5.
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout_1s) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
It was OK to change the variable name to 'inactive_timeout_1s' outside
of here, but within the subroutine, I don't think it is appropriate
because this is a parameter that potentially could have any value.
~
6.
+ # Give enough time to avoid multiple checkpoints
+ sleep($inactive_timeout_1s + 1);
+
+ # Run a checkpoint
+ $node->safe_psql('postgres', "CHECKPOINT");
Since you are not doing multiple checkpoints anymore, it looks like
that "Give enough time..." comment needs updating.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-27 03:09 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
@ 2024-11-27 10:54 ` Nisha Moond <[email protected]>
2024-11-27 21:37 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-11-27 23:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-11-28 07:59 ` RE: Introduce XID age and inactive timeout based replication slot invalidation Hayato Kuroda (Fujitsu) <[email protected]>
2024-11-28 14:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
0 siblings, 4 replies; 98+ messages in thread
From: Nisha Moond @ 2024-11-27 10:54 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: vignesh C <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Nov 27, 2024 at 8:39 AM Peter Smith <[email protected]> wrote:
>
> Hi Nisha,
>
> Here are some review comments for the patch v50-0002.
>
> ======
> src/backend/replication/slot.c
>
> InvalidatePossiblyObsoleteSlot:
>
> 1.
> + if (now &&
> + TimestampDifferenceExceeds(s->inactive_since, now,
> + replication_slot_inactive_timeout_sec * 1000))
>
> Previously this was using an additional call to SlotInactiveTimeoutCheckAllowed:
>
> + if (SlotInactiveTimeoutCheckAllowed(s) &&
> + TimestampDifferenceExceeds(s->inactive_since, now,
> + replication_slot_inactive_timeout * 1000))
>
> Is it OK to skip that call? e.g. can the slot fields possibly change
> between assigning the 'now' and acquiring the mutex? If not, then the
> current code is fine. The only reason for asking is because it is
> slightly suspicious that it was not done this "easy" way in the first
> place.
>
Good catch! While the mutex was being acquired right after the now
assignment, there was a rare chance of another process modifying the
slot in the meantime. So, I reverted the change in v51. To optimize
the SlotInactiveTimeoutCheckAllowed() call, it's sufficient to check
it here instead of during the 'now' assignment.
Attached v51 patch-set addressing all comments in [1] and [2].
[1] https://www.postgresql.org/message-id/CAHut%2BPtuiQj1hwm%3D73xJ8hWuw-9cXbN4dHJHpM6EXxubDJgmFA%40mail...
[2] https://www.postgresql.org/message-id/CAHut%2BPvi-g%2B9%2Bhjmjg44OzTN9L3YGQiCXBDAVaTVWvSn5SSwmw%40ma...
--
Thanks,
Nisha
Attachments:
[application/octet-stream] v51-0001-Add-error-handling-while-acquiring-a-replication.patch (7.9K, ../../CABdArM4QR7LPoJ_ES+_1miQZ9ziwsOUun4FGSDhPdrxsugH_0A@mail.gmail.com/2-v51-0001-Add-error-handling-while-acquiring-a-replication.patch)
download | inline diff:
From 2319f1bf761fe2dea1aa84068f400b64f045e315 Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Mon, 18 Nov 2024 16:13:26 +0530
Subject: [PATCH v51 1/2] Add error handling while acquiring a replication slot
In ReplicationSlotAcquire(), raise an error for invalid slots if the
caller specifies error_if_invalid=true.
---
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 4 +-
src/backend/replication/slot.c | 48 +++++++++++++++++--
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/include/replication/slot.h | 3 +-
src/test/recovery/t/019_replslot_limit.pl | 2 +-
8 files changed, 55 insertions(+), 12 deletions(-)
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index f4f80b2312..83d6e3811e 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -446,7 +446,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -665,7 +665,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 6828100cf1..8479d03c63 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -535,9 +535,12 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if error_if_invalid is true and the slot is found to
+ * be invalid.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +618,45 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot is found to
+ * be invalid.
+ */
+ if (error_if_invalid &&
+ s->data.invalidated != RS_INVAL_NONE)
+ {
+ StringInfoData err_detail;
+
+ initStringInfo(&err_detail);
+
+ switch (s->data.invalidated)
+ {
+ case RS_INVAL_WAL_REMOVED:
+ appendStringInfo(&err_detail, _("This slot has been invalidated because the required WAL has been removed."));
+ break;
+
+ case RS_INVAL_HORIZON:
+ appendStringInfo(&err_detail, _("This slot has been invalidated because the required rows have been removed."));
+ break;
+
+ case RS_INVAL_WAL_LEVEL:
+ appendStringInfo(&err_detail, _("This slot has been invalidated because \"%s\" is insufficient for slot."),
+ "wal_level");
+ break;
+
+ case RS_INVAL_NONE:
+ pg_unreachable();
+ }
+
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail_internal("%s", err_detail.data));
+
+ pfree(err_detail.data);
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -785,7 +827,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +854,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, false);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 488a161b3e..578cff64c8 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -536,7 +536,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 371eef3ddd..b36ae90b2c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -816,7 +816,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1434,7 +1434,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 8a45b5827e..3322848e03 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -298,7 +298,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index d2cf786fd5..f5f2d22163 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -253,7 +253,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index efb4ba3af1..333e040e7f 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -234,7 +234,7 @@ my $failed = 0;
for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
{
if ($node_standby->log_contains(
- "requested WAL segment [0-9A-F]+ has already been removed",
+ "This slot has been invalidated because the required WAL has been removed",
$logstart))
{
$failed = 1;
--
2.34.1
[application/octet-stream] v51-0002-Introduce-inactive_timeout-based-replication-slo.patch (27.2K, ../../CABdArM4QR7LPoJ_ES+_1miQZ9ziwsOUun4FGSDhPdrxsugH_0A@mail.gmail.com/3-v51-0002-Introduce-inactive_timeout-based-replication-slo.patch)
download | inline diff:
From 548b813ed7e2e7933e3f10a3938bb21d8a57a866 Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Wed, 27 Nov 2024 10:52:14 +0530
Subject: [PATCH v51 2/2] Introduce inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage per instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named replication_slot_inactive_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are inactive for longer than this amount of
time.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.
---
doc/src/sgml/config.sgml | 35 ++++
doc/src/sgml/system-views.sgml | 10 +-
src/backend/replication/logical/slotsync.c | 13 +-
src/backend/replication/slot.c | 159 +++++++++++++--
src/backend/utils/misc/guc_tables.c | 12 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 22 ++
src/include/utils/guc_hooks.h | 2 +
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 190 ++++++++++++++++++
10 files changed, 419 insertions(+), 26 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 76ab72db96..b8d094447b 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4585,6 +4585,41 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the inactive timeout invalidation mechanism. This parameter can only
+ be set in the <filename>postgresql.conf</filename> file or on the
+ server command line.
+ </para>
+
+ <para>
+ Slot invalidation due to inactive timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from the primary server (i.e., standby slots having
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+ value <literal>true</literal>).
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index a586156614..b90c163eb2 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2566,7 +2566,8 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</para>
<para>
The time when the slot became inactive. <literal>NULL</literal> if the
- slot is currently being streamed.
+ slot is currently being streamed. Once the slot is invalidated, this
+ value will remain unchanged until we shutdown the server.
Note that for slots on the standby that are being synced from a
primary server (whose <structfield>synced</structfield> field is
<literal>true</literal>), the <structfield>inactive_since</structfield>
@@ -2620,6 +2621,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has remained
+ inactive beyond the duration specified by the
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 83d6e3811e..9777c6a9cc 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
static void
update_synced_slots_inactive_since(void)
{
- TimestampTz now = 0;
+ TimestampTz now;
/*
* We need to update inactive_since only when we are promoting standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
/* The slot sync worker or SQL function mustn't be running by now */
Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (int i = 0; i < max_replication_slots; i++)
@@ -1537,13 +1540,7 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- /* Use the same inactive_since time for all the slots. */
- if (now == 0)
- now = GetCurrentTimestamp();
-
- SpinLockAcquire(&s->mutex);
- s->inactive_since = now;
- SpinLockRelease(&s->mutex);
+ ReplicationSlotSetInactiveSince(s, now, true);
}
}
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 8479d03c63..1e82b7fab6 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout_sec = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -644,6 +646,11 @@ retry:
"wal_level");
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ appendStringInfo(&err_detail, _("inactivity exceeded the time limit set by \"%s\"."),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -745,16 +752,12 @@ ReplicationSlotRelease(void)
*/
SpinLockAcquire(&slot->mutex);
slot->active_pid = 0;
- slot->inactive_since = now;
+ ReplicationSlotSetInactiveSince(slot, now, false);
SpinLockRelease(&slot->mutex);
ConditionVariableBroadcast(&slot->active_cv);
}
else
- {
- SpinLockAcquire(&slot->mutex);
- slot->inactive_since = now;
- SpinLockRelease(&slot->mutex);
- }
+ ReplicationSlotSetInactiveSince(slot, now, true);
MyReplicationSlot = NULL;
@@ -1550,7 +1553,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1580,6 +1584,15 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s, exceeding the time limit set by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1596,6 +1609,30 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
pfree(err_detail.data);
}
+/*
+ * Is inactive timeout invalidation possible for this replication slot?
+ *
+ * Inactive timeout invalidation is allowed only when:
+ *
+ * 1. Inactive timeout is set
+ * 2. Slot is inactive
+ * 3. The slot is not being synced from the primary while the server
+ * is in recovery
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+IsSlotInactiveTimeoutPossible(ReplicationSlot *s)
+{
+ return (replication_slot_inactive_timeout_sec > 0 &&
+ s->inactive_since > 0 &&
+ !(RecoveryInProgress() && s->data.synced));
+}
+
/*
* Helper for InvalidateObsoleteReplicationSlots
*
@@ -1623,6 +1660,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1630,6 +1668,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1640,6 +1679,15 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1693,6 +1741,20 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (IsSlotInactiveTimeoutPossible(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout_sec * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1718,11 +1780,13 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1777,7 +1841,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1823,7 +1888,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1846,6 +1912,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout has occurred
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1898,7 +1965,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1956,6 +2024,38 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
@@ -2250,6 +2350,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2410,6 +2511,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be \"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2440,9 +2544,11 @@ RestoreSlotFromDisk(const char *name)
/*
* Set the time since the slot has become inactive after loading the
* slot from the disk into memory. Whoever acquires the slot i.e.
- * makes the slot active will reset it.
+ * makes the slot active will reset it. Avoid calling
+ * ReplicationSlotSetInactiveSince() here, as it will not set the time
+ * for invalid slots.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;
restored = true;
break;
@@ -2845,3 +2951,22 @@ WaitForStandbyConfirmation(XLogRecPtr wait_for_lsn)
ConditionVariableCancelSleep();
}
+
+/*
+ * GUC check_hook for replication_slot_inactive_timeout
+ *
+ * The replication_slot_inactive_timeout must be disabled (set to 0)
+ * during the binary upgrade.
+ */
+bool
+check_replication_slot_inactive_timeout(int *newval, void **extra, GucSource source)
+{
+ if (IsBinaryUpgrade && *newval != 0)
+ {
+ GUC_check_errdetail("The value of \"%s\" must be set to 0 during binary upgrade mode.",
+ "replication_slot_inactive_timeout");
+ return false;
+ }
+
+ return true;
+}
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 9845abd693..264ebd59ad 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3038,6 +3038,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout_sec,
+ 0, 0, INT_MAX,
+ check_replication_slot_inactive_timeout, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 407cd1e08c..18596686da 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index f5f2d22163..7682f610ea 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -228,6 +230,25 @@ typedef struct ReplicationSlotCtlData
ReplicationSlot replication_slots[1];
} ReplicationSlotCtlData;
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz now,
+ bool acquire_lock)
+{
+ if (s->data.invalidated != RS_INVAL_NONE)
+ return;
+
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ s->inactive_since = now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
+
/*
* Pointers to shared memory
*/
@@ -237,6 +258,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout_sec;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
diff --git a/src/include/utils/guc_hooks.h b/src/include/utils/guc_hooks.h
index 5813dba0a2..e36b6cfe21 100644
--- a/src/include/utils/guc_hooks.h
+++ b/src/include/utils/guc_hooks.h
@@ -174,5 +174,7 @@ extern void assign_wal_sync_method(int new_wal_sync_method, void *extra);
extern bool check_synchronized_standby_slots(char **newval, void **extra,
GucSource source);
extern void assign_synchronized_standby_slots(const char *newval, void *extra);
+extern bool check_replication_slot_inactive_timeout(int *newval, void **extra,
+ GucSource source);
#endif /* GUC_HOOKS_H */
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..708a2a3798 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -51,6 +51,7 @@ tests += {
't/040_standby_failover_slots_sync.pl',
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
+ 't/050_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..861509f050
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,190 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start
+#
+# Test invalidation of streaming standby slot and logical failover slot on the
+# primary due to inactive timeout. Also, test logical failover slot synced to
+# the standby from the primary doesn't get invalidated on its own, but gets the
+# invalidated state from the primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot1'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
+);
+
+# Create standby slot on the primary
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot1', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until the standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+my $logstart = -s $standby1->logfile;
+
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $inactive_timeout_1s = 1;
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout_1s}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced
+ AND NOT temporary
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'logical slot sync_slot1 is synced to standby');
+
+# Give enough time for inactive_since to exceed the timeout
+sleep($inactive_timeout_1s + 1);
+
+# On standby, synced slots are not invalidated by the inactive timeout
+# until the invalidation state is propagated from the primary.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'check that synced slot sync_slot1 has not been invalidated on standby');
+
+$logstart = -s $primary->logfile;
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout_1s}s';
+]);
+$primary->reload;
+
+# Wait for logical failover slot to become inactive on the primary. Note that
+# nobody has acquired the slot yet, so it must get invalidated due to
+# inactive timeout.
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+ $inactive_timeout_1s);
+
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to inactive timeout. The standby must just
+# sync the invalidated state.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason = 'inactive_timeout';}
+ ),
+ "t",
+ 'check that invalidation of synced slot sync_slot1 is synced on standby');
+
+# Make the standby slot on the primary inactive and check for invalidation
+$standby1->stop;
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+ $inactive_timeout_1s);
+
+# Testcase end
+# =============================================================================
+
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+
+ trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout);
+
+ # Check that an invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
+ ok( $stderr =~ /can no longer get changes from replication slot "$slot"/,
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
+}
+
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
+
+ # Give enough time for inactive_since to exceed the timeout
+ sleep($inactive_timeout + 1);
+
+ # Run a checkpoint
+ $node->safe_psql('postgres', "CHECKPOINT");
+
+ # The slot's invalidation should be logged
+ $node->wait_for_log(qr/invalidating obsolete replication slot \"$slot\"/,
+ $offset);
+
+ # Check that the invalidation reason is 'inactive_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-27 03:09 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-11-27 10:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-27 21:37 ` Peter Smith <[email protected]>
3 siblings, 0 replies; 98+ messages in thread
From: Peter Smith @ 2024-11-27 21:37 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: vignesh C <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi Nisha, here are my review comments for the patch v51-0001.
======
src/backend/replication/slot.c
ReplicationSlotAcquire:
1.
+ ereport(ERROR,
+ errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail_internal("%s", err_detail.data));
+
+ pfree(err_detail.data);
+ }
+
Won't the 'pfree' be unreachable due to the prior ereport ERROR?
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-27 03:09 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-11-27 10:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-27 23:50 ` Peter Smith <[email protected]>
2024-11-29 12:37 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
3 siblings, 1 reply; 98+ messages in thread
From: Peter Smith @ 2024-11-27 23:50 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: vignesh C <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi Nisha. Here are some review comments for patch v51-0002.
======
doc/src/sgml/system-views.sgml
1.
The time when the slot became inactive. <literal>NULL</literal> if the
- slot is currently being streamed.
+ slot is currently being streamed. Once the slot is invalidated, this
+ value will remain unchanged until we shutdown the server.
.
I think "Once the ..." kind of makes it sound like invalidation is
inevitable. Also maybe it's better to remove the "we".
SUGGESTION:
If the slot becomes invalidated, this value will remain unchanged
until server shutdown.
======
src/backend/replication/slot.c
ReplicationSlotAcquire:
2.
GENERAL.
This just is a question/idea. It may not be feasible to change. It
seems like there is a lot of overlap between the error messages in
'ReplicationSlotAcquire' which are saying "This slot has been
invalidated because...", and with the other function
'ReportSlotInvalidation' which is kind of the same but called in
different circumstances and with slightly different message text. I
wondered if there is a way to use common code to unify these messages
instead of having a nearly duplicate set of messages for all the
invalidation causes?
~~~
3.
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ appendStringInfo(&err_detail, _("inactivity exceeded the time limit
set by \"%s\"."),
+ "replication_slot_inactive_timeout");
+ break;
Should this err_detail also say "This slot has been invalidated
because ..." like all the others?
~~~
InvalidatePossiblyObsoleteSlot:
4.
+ case RS_INVAL_INACTIVE_TIMEOUT:
+
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (IsSlotInactiveTimeoutPossible(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout_sec * 1000))
+ {
Maybe this code should have Assert(now > 0); before the condition just
as a way to 'document' that it is assumed 'now' was already set this
outside the mutex.
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-27 03:09 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-11-27 10:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-27 23:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
@ 2024-11-29 12:37 ` Nisha Moond <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Nisha Moond @ 2024-11-29 12:37 UTC (permalink / raw)
To: Peter Smith <[email protected]>; +Cc: vignesh C <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Nov 28, 2024 at 5:20 AM Peter Smith <[email protected]> wrote:
>
> Hi Nisha. Here are some review comments for patch v51-0002.
>
> ======
> src/backend/replication/slot.c
>
> ReplicationSlotAcquire:
>
> 2.
> GENERAL.
>
> This just is a question/idea. It may not be feasible to change. It
> seems like there is a lot of overlap between the error messages in
> 'ReplicationSlotAcquire' which are saying "This slot has been
> invalidated because...", and with the other function
> 'ReportSlotInvalidation' which is kind of the same but called in
> different circumstances and with slightly different message text. I
> wondered if there is a way to use common code to unify these messages
> instead of having a nearly duplicate set of messages for all the
> invalidation causes?
>
The error handling could be moved to a new function; however, as you
pointed out, the contexts in which these functions are called differ.
IMO, a single error message may not suit both cases. For example,
ReportSlotInvalidation provides additional details and a hint in its
message, which isn’t necessary for ReplicationSlotAcquire.
Thoughts?
--
Thanks,
Nisha
^ permalink raw reply [nested|flat] 98+ messages in thread
* RE: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-27 03:09 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-11-27 10:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-28 07:59 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-11-29 12:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
3 siblings, 1 reply; 98+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2024-11-28 07:59 UTC (permalink / raw)
To: 'Nisha Moond' <[email protected]>; +Cc: vignesh C <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; Peter Smith <[email protected]>
Dear Nisha,
>
> Attached v51 patch-set addressing all comments in [1] and [2].
>
Thanks for working on the feature! I've stated to review the patch.
Here are my comments - sorry if there are something which have already been discussed.
The thread is too long to follow correctly.
Comments for 0001
=============
01. binary_upgrade_logical_slot_has_caught_up
ISTM that error_if_invalid is set to true when the slot can be moved forward, otherwise
it is set to false. Regarding the binary_upgrade_logical_slot_has_caught_up, however,
only valid slots will be passed to the funciton (see pg_upgrade/info.c) so I feel
it is OK to set to true. Thought?
02. ReplicationSlotAcquire
According to other functions, we are adding to a note to the translator when
parameters represent some common nouns, GUC names. I feel we should add a comment
for RS_INVAL_WAL_LEVEL part based on it.
Comments for 0002
=============
03. check_replication_slot_inactive_timeout
Can we overwrite replication_slot_inactive_timeout to zero when pg_uprade (and also
pg_createsubscriber?) starts a server process? Several parameters have already been
specified via -c option at that time. This can avoid an error while the upgrading.
Note that this part is still needed even if you accept the comment. Users can
manually boot with upgrade mode.
04. ReplicationSlotAcquire
Same comment as 02.
05. ReportSlotInvalidation
Same comment as 02.
06. found bug
While testing the patch, I found that slots can be invalidated too early when when
the GUC is quite large. I think because an overflow is caused in InvalidatePossiblyObsoleteSlot().
- Reproducer
I set the replication_slot_inactive_timeout to INT_MAX and executed below commands,
and found that the slot is invalidated.
```
postgres=# SHOW replication_slot_inactive_timeout;
replication_slot_inactive_timeout
-----------------------------------
2147483647s
(1 row)
postgres=# SELECT * FROM pg_create_logical_replication_slot('test', 'test_decoding');
slot_name | lsn
-----------+-----------
test | 0/18B7F38
(1 row)
postgres=# CHECKPOINT ;
CHECKPOINT
postgres=# SELECT slot_name, inactive_since, invalidation_reason FROM pg_replication_slots ;
slot_name | inactive_since | invalidation_reason
-----------+-------------------------------+---------------------
test | 2024-11-28 07:50:25.927594+00 | inactive_timeout
(1 row)
```
- analysis
In InvalidatePossiblyObsoleteSlot(), replication_slot_inactive_timeout_sec * 1000
is passed to the third argument of TimestampDifferenceExceeds(), which is also the
integer datatype. This causes an overflow and parameter is handled as the small
value.
- solution
I think there are two possible solutions. You can choose one of them:
a. Make the maximum INT_MAX/1000, or
b. Change the unit to millisecond.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-27 03:09 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-11-27 10:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-28 07:59 ` RE: Introduce XID age and inactive timeout based replication slot invalidation Hayato Kuroda (Fujitsu) <[email protected]>
@ 2024-11-29 12:36 ` Nisha Moond <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Nisha Moond @ 2024-11-29 12:36 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <[email protected]>; +Cc: vignesh C <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; Peter Smith <[email protected]>
On Thu, Nov 28, 2024 at 1:29 PM Hayato Kuroda (Fujitsu)
<[email protected]> wrote:
>
> Dear Nisha,
>
> >
> > Attached v51 patch-set addressing all comments in [1] and [2].
> >
>
> Thanks for working on the feature! I've stated to review the patch.
> Here are my comments - sorry if there are something which have already been discussed.
> The thread is too long to follow correctly.
>
> Comments for 0001
> =============
>
> 01. binary_upgrade_logical_slot_has_caught_up
>
> ISTM that error_if_invalid is set to true when the slot can be moved forward, otherwise
> it is set to false. Regarding the binary_upgrade_logical_slot_has_caught_up, however,
> only valid slots will be passed to the funciton (see pg_upgrade/info.c) so I feel
> it is OK to set to true. Thought?
>
Right, corrected the call with error_if_invalid as true.
> Comments for 0002
> =============
>
> 03. check_replication_slot_inactive_timeout
>
> Can we overwrite replication_slot_inactive_timeout to zero when pg_uprade (and also
> pg_createsubscriber?) starts a server process? Several parameters have already been
> specified via -c option at that time. This can avoid an error while the upgrading.
> Note that this part is still needed even if you accept the comment. Users can
> manually boot with upgrade mode.
>
Done.
> 06. found bug
>
> While testing the patch, I found that slots can be invalidated too early when when
> the GUC is quite large. I think because an overflow is caused in InvalidatePossiblyObsoleteSlot().
>
> - Reproducer
>
> I set the replication_slot_inactive_timeout to INT_MAX and executed below commands,
> and found that the slot is invalidated.
>
> ```
> postgres=# SHOW replication_slot_inactive_timeout;
> replication_slot_inactive_timeout
> -----------------------------------
> 2147483647s
> (1 row)
> postgres=# SELECT * FROM pg_create_logical_replication_slot('test', 'test_decoding');
> slot_name | lsn
> -----------+-----------
> test | 0/18B7F38
> (1 row)
> postgres=# CHECKPOINT ;
> CHECKPOINT
> postgres=# SELECT slot_name, inactive_since, invalidation_reason FROM pg_replication_slots ;
> slot_name | inactive_since | invalidation_reason
> -----------+-------------------------------+---------------------
> test | 2024-11-28 07:50:25.927594+00 | inactive_timeout
> (1 row)
> ```
>
> - analysis
>
> In InvalidatePossiblyObsoleteSlot(), replication_slot_inactive_timeout_sec * 1000
> is passed to the third argument of TimestampDifferenceExceeds(), which is also the
> integer datatype. This causes an overflow and parameter is handled as the small
> value.
>
> - solution
>
> I think there are two possible solutions. You can choose one of them:
>
> a. Make the maximum INT_MAX/1000, or
> b. Change the unit to millisecond.
>
Fixed. It is reasonable to align with other timeout parameters by
using milliseconds as the unit.
--
Thanks,
Nisha
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:12 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-20 07:59 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-21 12:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-27 03:09 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-11-27 10:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-28 14:10 ` vignesh C <[email protected]>
3 siblings, 0 replies; 98+ messages in thread
From: vignesh C @ 2024-11-28 14:10 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: Peter Smith <[email protected]>; shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, 27 Nov 2024 at 16:25, Nisha Moond <[email protected]> wrote:
>
> On Wed, Nov 27, 2024 at 8:39 AM Peter Smith <[email protected]> wrote:
> >
> > Hi Nisha,
> >
> > Here are some review comments for the patch v50-0002.
> >
> > ======
> > src/backend/replication/slot.c
> >
> > InvalidatePossiblyObsoleteSlot:
> >
> > 1.
> > + if (now &&
> > + TimestampDifferenceExceeds(s->inactive_since, now,
> > + replication_slot_inactive_timeout_sec * 1000))
> >
> > Previously this was using an additional call to SlotInactiveTimeoutCheckAllowed:
> >
> > + if (SlotInactiveTimeoutCheckAllowed(s) &&
> > + TimestampDifferenceExceeds(s->inactive_since, now,
> > + replication_slot_inactive_timeout * 1000))
> >
> > Is it OK to skip that call? e.g. can the slot fields possibly change
> > between assigning the 'now' and acquiring the mutex? If not, then the
> > current code is fine. The only reason for asking is because it is
> > slightly suspicious that it was not done this "easy" way in the first
> > place.
> >
> Good catch! While the mutex was being acquired right after the now
> assignment, there was a rare chance of another process modifying the
> slot in the meantime. So, I reverted the change in v51. To optimize
> the SlotInactiveTimeoutCheckAllowed() call, it's sufficient to check
> it here instead of during the 'now' assignment.
>
> Attached v51 patch-set addressing all comments in [1] and [2].
Few comments:
1) replication_slot_inactive_timeout can be mentioned in logical
replication config, we could mention something like:
Logical replication slot is also affected by replication_slot_inactive_timeout
2.a) Is this change applicable only for inactive timeout or it is
applicable to others like wal removed, wal level etc also? If it is
applicable to all of them we could move this to the first patch and
update the commit message:
+ * If the slot can be acquired, do so and mark it as
invalidated. If
+ * the slot is already ours, mark it as invalidated.
Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
2.b) Also this MyReplicationSlot and active_pid check can be in same line:
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
3) Error detail should start in upper case here similar to how others are done:
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ appendStringInfo(&err_detail,
_("inactivity exceeded the time limit set by \"%s\"."),
+
"replication_slot_inactive_timeout");
+ break;
4) Since this change is not related to this patch, we can move this to
the first patch and update the commit message:
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data,
size_t startup_data_len)
static void
update_synced_slots_inactive_since(void)
{
- TimestampTz now = 0;
+ TimestampTz now;
/*
* We need to update inactive_since only when we are promoting
standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
/* The slot sync worker or SQL function mustn't be running by now */
Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
5) Since this change is not related to this patch, we can move this to
the first patch.
@@ -2250,6 +2350,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2410,6 +2511,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be
\"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2440,9 +2544,11 @@ RestoreSlotFromDisk(const char *name)
/*
* Set the time since the slot has become inactive
after loading the
* slot from the disk into memory. Whoever acquires
the slot i.e.
- * makes the slot active will reset it.
+ * makes the slot active will reset it. Avoid calling
+ * ReplicationSlotSetInactiveSince() here, as it will
not set the time
+ * for invalid slots.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ slot->inactive_since = now;
[1] - https://www.postgresql.org/docs/current/logical-replication-config.html
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
@ 2024-11-19 07:20 ` Nisha Moond <[email protected]>
2024-11-21 05:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: Nisha Moond @ 2024-11-19 07:20 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Nov 14, 2024 at 9:14 AM vignesh C <[email protected]> wrote:
>
> On Wed, 13 Nov 2024 at 15:00, Nisha Moond <[email protected]> wrote:
> >
> > Please find the v48 patch attached.
> >
> > On Thu, Sep 19, 2024 at 9:40 AM shveta malik <[email protected]> wrote:
> > >
> > > When we promote hot standby with synced logical slots to become new
> > > primary, the logical slots are never invalidated with
> > > 'inactive_timeout' on new primary. It seems the check in
> > > SlotInactiveTimeoutCheckAllowed() is wrong. We should allow
> > > invalidation of slots on primary even if they are marked as 'synced'.
> >
> > fixed.
> >
> > > I have raised 4 issues so far on v46, the first 3 are in [1],[2],[3].
> > > Once all these are addressed, I can continue reviewing further.
> > >
> >
> > Fixed issues reported in [1], [2].
>
> Few comments:
Thanks for the review.
>
> 2) Currently it allows a minimum value of less than 1 second like in
> milliseconds, I feel we can have some minimum value at least something
> like checkpoint_timeout:
> diff --git a/src/backend/utils/misc/guc_tables.c
> b/src/backend/utils/misc/guc_tables.c
> index 8a67f01200..367f510118 100644
> --- a/src/backend/utils/misc/guc_tables.c
> +++ b/src/backend/utils/misc/guc_tables.c
> @@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
> NULL, NULL, NULL
> },
>
> + {
> + {"replication_slot_inactive_timeout", PGC_SIGHUP,
> REPLICATION_SENDING,
> + gettext_noop("Sets the amount of time a
> replication slot can remain inactive before "
> + "it will be invalidated."),
> + NULL,
> + GUC_UNIT_S
> + },
> + &replication_slot_inactive_timeout,
> + 0, 0, INT_MAX,
> + NULL, NULL, NULL
> + },
>
Currently, the feature is disabled by default when
replication_slot_inactive_timeout = 0. However, if we set a minimum
value, the default_val cannot be less than min_val, making it
impossible to use 0 to disable the feature.
Thoughts or any suggestions?
>
> 4) I'm not sure if this change required by this patch or is it a
> general optimization, if it is required for this patch we can detail
> the comments:
> @@ -2208,6 +2328,7 @@ RestoreSlotFromDisk(const char *name)
> bool restored = false;
> int readBytes;
> pg_crc32c checksum;
> + TimestampTz now;
>
> /* no need to lock here, no concurrent access allowed yet */
>
> @@ -2368,6 +2489,9 @@ RestoreSlotFromDisk(const char *name)
> NameStr(cp.slotdata.name)),
> errhint("Change \"wal_level\" to be
> \"replica\" or higher.")));
>
> + /* Use same inactive_since time for all slots */
> + now = GetCurrentTimestamp();
> +
> /* nothing can be active yet, don't lock anything */
> for (i = 0; i < max_replication_slots; i++)
> {
> @@ -2400,7 +2524,7 @@ RestoreSlotFromDisk(const char *name)
> * slot from the disk into memory. Whoever acquires
> the slot i.e.
> * makes the slot active will reset it.
> */
> - slot->inactive_since = GetCurrentTimestamp();
> + slot->inactive_since = now;
>
After removing the "ReplicationSlotSetInactiveSince" from here, it
became irrelevant to this patch. Now, it is a general optimization to
set the same timestamp for all slots while restoring from disk. I have
added a few comments as per Peter's suggestion.
> 5) Why should the slot invalidation be updated during shutdown,
> shouldn't the inactive_since value be intact during shutdown?
> - <literal>NULL</literal> if the slot is currently being used.
> - Note that for slots on the standby that are being synced from a
> + <literal>NULL</literal> if the slot is currently being used. Once the
> + slot is invalidated, this value will remain unchanged until we shutdown
> + the server. Note that for slots on the standby that are being
> synced from a
>
The "inactive_since" data of a slot is not stored on disk, so the
older value cannot be restored after a restart.
--
Thanks,
Nisha
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-19 04:10 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-11-13 09:30 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
2024-11-14 03:44 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
2024-11-19 07:20 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-21 05:06 ` vignesh C <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: vignesh C @ 2024-11-21 05:06 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: shveta malik <[email protected]>; Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, 19 Nov 2024 at 12:51, Nisha Moond <[email protected]> wrote:
>
> On Thu, Nov 14, 2024 at 9:14 AM vignesh C <[email protected]> wrote:
> >
> > On Wed, 13 Nov 2024 at 15:00, Nisha Moond <[email protected]> wrote:
> > >
> > > Please find the v48 patch attached.
> > >
> > 2) Currently it allows a minimum value of less than 1 second like in
> > milliseconds, I feel we can have some minimum value at least something
> > like checkpoint_timeout:
> > diff --git a/src/backend/utils/misc/guc_tables.c
> > b/src/backend/utils/misc/guc_tables.c
> > index 8a67f01200..367f510118 100644
> > --- a/src/backend/utils/misc/guc_tables.c
> > +++ b/src/backend/utils/misc/guc_tables.c
> > @@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
> > NULL, NULL, NULL
> > },
> >
> > + {
> > + {"replication_slot_inactive_timeout", PGC_SIGHUP,
> > REPLICATION_SENDING,
> > + gettext_noop("Sets the amount of time a
> > replication slot can remain inactive before "
> > + "it will be invalidated."),
> > + NULL,
> > + GUC_UNIT_S
> > + },
> > + &replication_slot_inactive_timeout,
> > + 0, 0, INT_MAX,
> > + NULL, NULL, NULL
> > + },
> >
>
> Currently, the feature is disabled by default when
> replication_slot_inactive_timeout = 0. However, if we set a minimum
> value, the default_val cannot be less than min_val, making it
> impossible to use 0 to disable the feature.
> Thoughts or any suggestions?
We could implement this similarly to how the vacuum_buffer_usage_limit
GUC is handled. Setting the value to 0 would allow the operation to
use any amount of shared_buffers. Otherwise, valid sizes would range
from 128 kB to 16 GB. Similarly, we can modify
check_replication_slot_inactive_timeout to behave in the same way as
check_vacuum_buffer_usage_limit function.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 09:19 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-18 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-11-11 12:42 ` Nisha Moond <[email protected]>
1 sibling, 0 replies; 98+ messages in thread
From: Nisha Moond @ 2024-11-11 12:42 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Sep 18, 2024 at 3:31 PM shveta malik <[email protected]> wrote:
>
> On Wed, Sep 18, 2024 at 2:49 PM shveta malik <[email protected]> wrote:
> >
> > > > Please find the attached v46 patch having changes for the above review
> > > > comments and your test review comments and Shveta's review comments.
> > > >
>
> When the synced slot is marked as 'inactive_timeout' invalidated on
> hot standby due to invalidation of publisher 's failover slot, the
> former starts showing NULL' inactive_since'. Is this intentional
> behaviour? I feel inactive_since should be non-NULL here too?
> Thoughts?
>
> physical standby:
> postgres=# select slot_name, inactive_since, invalidation_reason,
> failover, synced from pg_replication_slots;
> slot_name | inactive_since |
> invalidation_reason | failover | synced
> -------------+----------------------------------+---------------------+----------+--------
> sub2 | 2024-09-18 15:20:04.364998+05:30 | | t | t
> sub3 | 2024-09-18 15:20:04.364953+05:30 | | t | t
>
> After sync of invalidation_reason:
>
> slot_name | inactive_since | invalidation_reason |
> failover | synced
> -------------+----------------------------------+---------------------+----------+--------
> sub2 | | inactive_timeout | t | t
> sub3 | | inactive_timeout | t | t
>
>
For synced slots on the standby, inactive_since indicates the last
synchronization time rather than the time the slot became inactive
(see doc - https://www.postgresql.org/docs/devel/view-pg-replication-slots.html).
In the reported case above, once a synced slot is invalidated we don't
even keep the last synchronization time for it. This is because when a
synced slot on the standby is marked invalid, inactive_since is reset
to NULL each time the slot-sync worker acquires a lock on it. This
lock acquisition before checking invalidation is done to avoid certain
race conditions and will activate the slot temporarily, resetting
inactive_since. Later, the slot-sync worker updates inactive_since for
all synced slots to the current synchronization time. However, for
invalid slots, this update is skipped, as per the patch’s design.
If we want to preserve the inactive_since value for the invalid synced
slots on standby, we need to clarify the time it should display. Here
are three possible approaches:
1) Copy the primary's inactive_since upon invalidation: When a slot
becomes invalid on the primary, the slot-sync worker could copy the
primary slot’s inactive_since to the standby slot and retain it, by
preventing future updates on the standby.
2) Use the current time of standby when the synced slot is marked
invalid for the first time and do not update it in subsequent sync
cycles if the slot is invalid.
Approach (2) seems more reasonable to me, however, Both 1) & 2)
approaches contradicts the purpose of inactive_since, as it no longer
represents either the true "last sync time" or the "time slot became
inactive" because the slot-sync worker acquires locks periodically for
syncing, and keeps activating the slot.
3) Continuously update inactive_since for invalid synced slots as
well: Treat invalid synced slots like valid ones by updating
inactive_since with each sync cycle. This way, we can keep the "last
sync time" in the inactive_since. However, this could confuse users
when "invalidation_reason=inactive_timeout" is set for a synced slot
on standby but inactive_since would reflect sync time rather than the
time slot became inactive. IIUC, on the primary, when
invalidation_reason=inactive_timeout for a slot, the inactive_since
represents the actual time the slot became inactive before getting
invalidated, unless the primary is restarted.
Thoughts?
--
Thanks,
Nisha
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-18 06:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-11-13 09:35 ` Nisha Moond <[email protected]>
1 sibling, 0 replies; 98+ messages in thread
From: Nisha Moond @ 2024-11-13 09:35 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Sep 18, 2024 at 12:22 PM shveta malik <[email protected]> wrote:
>
> On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
> <[email protected]> wrote:
> >
> > Hi,
> >
> >
> > Please find the attached v46 patch having changes for the above review
> > comments and your test review comments and Shveta's review comments.
> >
>
> Thanks for addressing comments.
>
> Is there a reason that we don't support this invalidation on hot
> standby for non-synced slots? Shouldn't we support this time-based
> invalidation there too just like other invalidations?
>
I don’t see any reason to *not* support this invalidation on hot
standby for non-synced slots. Therefore, I’ve added the same in v48.
--
Thanks,
Nisha
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-11-07 10:03 ` Nisha Moond <[email protected]>
2024-11-13 09:29 ` Re: Introduce XID age and inactive timeout based replication slot invalidation vignesh C <[email protected]>
3 siblings, 1 reply; 98+ messages in thread
From: Nisha Moond @ 2024-11-07 10:03 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
<[email protected]> wrote:
>
> Please find the attached v46 patch having changes for the above review
> comments and your test review comments and Shveta's review comments.
>
Hi,
I’ve reviewed this thread and am interested in working on the
remaining tasks and comments, as well as the future review comments.
However, Bharath, please let me know if you'd prefer to continue with
it.
Attached the rebased v47 patch, which also addresses Peter’s comments
#2, #3, and #4 at [1]. I will try addressing other comments as well in
next versions.
[1] https://www.postgresql.org/message-id/CAHut%2BPs%3Dx%2B2Hq5ue0YppOeDZqgHTnyw%3Du%2Bvs-qy0JRjKaeJtew%...
--
Thanks,
Nisha
Attachments:
[application/octet-stream] v47-0001-Introduce-inactive_timeout-based-replication-slo.patch (33.1K, ../../CABdArM7Ywqvj4_T-CLMhdxhrTUvWJb=Jcq-zC01av1TPKVxcLw@mail.gmail.com/2-v47-0001-Introduce-inactive_timeout-based-replication-slo.patch)
download | inline diff:
From 251dd5cb5502b438e40fe636486a4e1a5adbbc7a Mon Sep 17 00:00:00 2001
From: Nisha Moond <[email protected]>
Date: Thu, 7 Nov 2024 12:09:47 +0530
Subject: [PATCH v47] Introduce inactive_timeout based replication slot
invalidation
Till now, postgres has the ability to invalidate inactive
replication slots based on the amount of WAL (set via
max_slot_wal_keep_size GUC) that will be needed for the slots in
case they become active. However, choosing a default value for
this GUC is a bit tricky. Because the amount of WAL a database
generates, and the allocated storage for instance will vary
greatly in production, making it difficult to pin down a
one-size-fits-all value.
It is often easy for users to set a timeout of say 1 or 2 or n
days, after which all the inactive slots get invalidated. This
commit introduces a GUC named replication_slot_inactive_timeout.
When set, postgres invalidates slots (during non-shutdown
checkpoints) that are inactive for longer than this amount of
time.
Note that the inactive timeout invalidation mechanism is not
applicable for slots on the standby server that are being synced
from the primary server (i.e., standby slots having 'synced' field
'true'). Synced slots are always considered to be inactive because
they don't perform logical decoding to produce changes.
---
doc/src/sgml/config.sgml | 35 +++
doc/src/sgml/system-views.sgml | 7 +
.../replication/logical/logicalfuncs.c | 2 +-
src/backend/replication/logical/slotsync.c | 17 +-
src/backend/replication/slot.c | 163 +++++++++--
src/backend/replication/slotfuncs.c | 2 +-
src/backend/replication/walsender.c | 4 +-
src/backend/utils/adt/pg_upgrade_support.c | 2 +-
src/backend/utils/misc/guc_tables.c | 12 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/replication/slot.h | 25 +-
src/test/recovery/meson.build | 1 +
src/test/recovery/t/050_invalidate_slots.pl | 267 ++++++++++++++++++
13 files changed, 503 insertions(+), 35 deletions(-)
create mode 100644 src/test/recovery/t/050_invalidate_slots.pl
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d54f904956..f19abe91b9 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -4583,6 +4583,41 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-replication-slot-inactive-timeout" xreflabel="replication_slot_inactive_timeout">
+ <term><varname>replication_slot_inactive_timeout</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>replication_slot_inactive_timeout</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Invalidate replication slots that are inactive for longer than this
+ amount of time. If this value is specified without units,
+ it is taken as seconds. A value of zero (which is default) disables
+ the inactive timeout invalidation mechanism. This parameter can only
+ be set in the <filename>postgresql.conf</filename> file or on the
+ server command line.
+ </para>
+
+ <para>
+ Slot invalidation due to inactive timeout occurs during checkpoint.
+ The duration of slot inactivity is calculated using the slot's
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>inactive_since</structfield>
+ value.
+ </para>
+
+ <para>
+ Note that the inactive timeout invalidation mechanism is not
+ applicable for slots on the standby server that are being synced
+ from the primary server (i.e., standby slots having
+ <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
+ value <literal>true</literal>).
+ Synced slots are always considered to be inactive because they don't
+ perform logical decoding to produce changes.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-track-commit-timestamp" xreflabel="track_commit_timestamp">
<term><varname>track_commit_timestamp</varname> (<type>boolean</type>)
<indexterm>
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 61d28e701f..c909ef5bf1 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2618,6 +2618,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
perform logical decoding. It is set only for logical slots.
</para>
</listitem>
+ <listitem>
+ <para>
+ <literal>inactive_timeout</literal> means that the slot has been
+ inactive for longer than the amount of time specified by the
+ <xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
+ </para>
+ </listitem>
</itemizedlist>
</para></entry>
</row>
diff --git a/src/backend/replication/logical/logicalfuncs.c b/src/backend/replication/logical/logicalfuncs.c
index b4dd5cce75..56fc1a45a9 100644
--- a/src/backend/replication/logical/logicalfuncs.c
+++ b/src/backend/replication/logical/logicalfuncs.c
@@ -197,7 +197,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool confirm, bool bin
else
end_of_wal = GetXLogReplayRecPtr(NULL);
- ReplicationSlotAcquire(NameStr(*name), true);
+ ReplicationSlotAcquire(NameStr(*name), true, true);
PG_TRY();
{
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index d62186a510..4443bd53b4 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -446,7 +446,7 @@ drop_local_obsolete_slots(List *remote_slot_list)
if (synced_slot)
{
- ReplicationSlotAcquire(NameStr(local_slot->data.name), true);
+ ReplicationSlotAcquire(NameStr(local_slot->data.name), true, false);
ReplicationSlotDropAcquired();
}
@@ -665,7 +665,7 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* pre-check to ensure that at least one of the slot properties is
* changed before acquiring the slot.
*/
- ReplicationSlotAcquire(remote_slot->name, true);
+ ReplicationSlotAcquire(remote_slot->name, true, false);
Assert(slot == MyReplicationSlot);
@@ -1508,7 +1508,7 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
static void
update_synced_slots_inactive_since(void)
{
- TimestampTz now = 0;
+ TimestampTz now;
/*
* We need to update inactive_since only when we are promoting standby to
@@ -1523,6 +1523,9 @@ update_synced_slots_inactive_since(void)
/* The slot sync worker or SQL function mustn't be running by now */
Assert((SlotSyncCtx->pid == InvalidPid) && !SlotSyncCtx->syncing);
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
for (int i = 0; i < max_replication_slots; i++)
@@ -1537,13 +1540,7 @@ update_synced_slots_inactive_since(void)
/* The slot must not be acquired by any process */
Assert(s->active_pid == 0);
- /* Use the same inactive_since time for all the slots. */
- if (now == 0)
- now = GetCurrentTimestamp();
-
- SpinLockAcquire(&s->mutex);
- s->inactive_since = now;
- SpinLockRelease(&s->mutex);
+ ReplicationSlotSetInactiveSince(s, &now, true);
}
}
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 6828100cf1..0076e4b5ea 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -107,10 +107,11 @@ const char *const SlotInvalidationCauses[] = {
[RS_INVAL_WAL_REMOVED] = "wal_removed",
[RS_INVAL_HORIZON] = "rows_removed",
[RS_INVAL_WAL_LEVEL] = "wal_level_insufficient",
+ [RS_INVAL_INACTIVE_TIMEOUT] = "inactive_timeout",
};
/* Maximum number of invalidation causes */
-#define RS_INVAL_MAX_CAUSES RS_INVAL_WAL_LEVEL
+#define RS_INVAL_MAX_CAUSES RS_INVAL_INACTIVE_TIMEOUT
StaticAssertDecl(lengthof(SlotInvalidationCauses) == (RS_INVAL_MAX_CAUSES + 1),
"array length mismatch");
@@ -140,6 +141,7 @@ ReplicationSlot *MyReplicationSlot = NULL;
/* GUC variables */
int max_replication_slots = 10; /* the maximum number of replication
* slots */
+int replication_slot_inactive_timeout = 0;
/*
* This GUC lists streaming replication standby server slot names that
@@ -535,9 +537,12 @@ ReplicationSlotName(int index, Name name)
*
* An error is raised if nowait is true and the slot is currently in use. If
* nowait is false, we sleep until the slot is released by the owning process.
+ *
+ * An error is raised if error_if_invalid is true and the slot has been
+ * invalidated previously.
*/
void
-ReplicationSlotAcquire(const char *name, bool nowait)
+ReplicationSlotAcquire(const char *name, bool nowait, bool error_if_invalid)
{
ReplicationSlot *s;
int active_pid;
@@ -615,6 +620,22 @@ retry:
/* We made this slot active, so it's ours now. */
MyReplicationSlot = s;
+ /*
+ * An error is raised if error_if_invalid is true and the slot has been
+ * previously invalidated due to inactive timeout.
+ */
+ if (error_if_invalid &&
+ s->data.invalidated == RS_INVAL_INACTIVE_TIMEOUT)
+ {
+ Assert(s->inactive_since > 0);
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg("can no longer get changes from replication slot \"%s\"",
+ NameStr(s->data.name)),
+ errdetail("This slot has been invalidated because it was inactive for longer than the amount of time specified by \"%s\".",
+ "replication_slot_inactive_timeout")));
+ }
+
/*
* The call to pgstat_acquire_replslot() protects against stats for a
* different slot, from before a restart or such, being present during
@@ -703,16 +724,12 @@ ReplicationSlotRelease(void)
*/
SpinLockAcquire(&slot->mutex);
slot->active_pid = 0;
- slot->inactive_since = now;
+ ReplicationSlotSetInactiveSince(slot, &now, false);
SpinLockRelease(&slot->mutex);
ConditionVariableBroadcast(&slot->active_cv);
}
else
- {
- SpinLockAcquire(&slot->mutex);
- slot->inactive_since = now;
- SpinLockRelease(&slot->mutex);
- }
+ ReplicationSlotSetInactiveSince(slot, &now, true);
MyReplicationSlot = NULL;
@@ -785,7 +802,7 @@ ReplicationSlotDrop(const char *name, bool nowait)
{
Assert(MyReplicationSlot == NULL);
- ReplicationSlotAcquire(name, nowait);
+ ReplicationSlotAcquire(name, nowait, false);
/*
* Do not allow users to drop the slots which are currently being synced
@@ -812,7 +829,7 @@ ReplicationSlotAlter(const char *name, const bool *failover,
Assert(MyReplicationSlot == NULL);
Assert(failover || two_phase);
- ReplicationSlotAcquire(name, false);
+ ReplicationSlotAcquire(name, false, false);
if (SlotIsPhysical(MyReplicationSlot))
ereport(ERROR,
@@ -1508,7 +1525,8 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
NameData slotname,
XLogRecPtr restart_lsn,
XLogRecPtr oldestLSN,
- TransactionId snapshotConflictHorizon)
+ TransactionId snapshotConflictHorizon,
+ TimestampTz inactive_since)
{
StringInfoData err_detail;
bool hint = false;
@@ -1538,6 +1556,15 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
case RS_INVAL_WAL_LEVEL:
appendStringInfoString(&err_detail, _("Logical decoding on standby requires \"wal_level\" >= \"logical\" on the primary server."));
break;
+
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ Assert(inactive_since > 0);
+ appendStringInfo(&err_detail,
+ _("The slot has been inactive since %s for longer than the amount of time specified by \"%s\"."),
+ timestamptz_to_str(inactive_since),
+ "replication_slot_inactive_timeout");
+ break;
+
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1554,6 +1581,31 @@ ReportSlotInvalidation(ReplicationSlotInvalidationCause cause,
pfree(err_detail.data);
}
+/*
+ * Is this replication slot allowed for inactive timeout invalidation check?
+ *
+ * Inactive timeout invalidation is allowed only when:
+ *
+ * 1. Inactive timeout is set
+ * 2. Slot is inactive
+ * 3. Server is not in recovery
+ * 4. Slot is not being synced from the primary
+ *
+ * Note that the inactive timeout invalidation mechanism is not
+ * applicable for slots on the standby server that are being synced
+ * from the primary server (i.e., standby slots having 'synced' field 'true').
+ * Synced slots are always considered to be inactive because they don't
+ * perform logical decoding to produce changes.
+ */
+static inline bool
+SlotInactiveTimeoutCheckAllowed(ReplicationSlot *s)
+{
+ return (replication_slot_inactive_timeout > 0 &&
+ s->inactive_since > 0 &&
+ !RecoveryInProgress() &&
+ !s->data.synced);
+}
+
/*
* Helper for InvalidateObsoleteReplicationSlots
*
@@ -1581,6 +1633,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
TransactionId initial_catalog_effective_xmin = InvalidTransactionId;
XLogRecPtr initial_restart_lsn = InvalidXLogRecPtr;
ReplicationSlotInvalidationCause invalidation_cause_prev PG_USED_FOR_ASSERTS_ONLY = RS_INVAL_NONE;
+ TimestampTz inactive_since = 0;
for (;;)
{
@@ -1588,6 +1641,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
NameData slotname;
int active_pid = 0;
ReplicationSlotInvalidationCause invalidation_cause = RS_INVAL_NONE;
+ TimestampTz now = 0;
Assert(LWLockHeldByMeInMode(ReplicationSlotControlLock, LW_SHARED));
@@ -1598,6 +1652,16 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
break;
}
+ if (cause == RS_INVAL_INACTIVE_TIMEOUT &&
+ SlotInactiveTimeoutCheckAllowed(s))
+ {
+ /*
+ * We get the current time beforehand to avoid system call while
+ * holding the spinlock.
+ */
+ now = GetCurrentTimestamp();
+ }
+
/*
* Check if the slot needs to be invalidated. If it needs to be
* invalidated, and is not currently acquired, acquire it and mark it
@@ -1651,6 +1715,25 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
if (SlotIsLogical(s))
invalidation_cause = cause;
break;
+ case RS_INVAL_INACTIVE_TIMEOUT:
+ /*
+ * Check if the slot needs to be invalidated due to
+ * replication_slot_inactive_timeout GUC.
+ */
+ if (SlotInactiveTimeoutCheckAllowed(s) &&
+ TimestampDifferenceExceeds(s->inactive_since, now,
+ replication_slot_inactive_timeout * 1000))
+ {
+ invalidation_cause = cause;
+ inactive_since = s->inactive_since;
+
+ /*
+ * Invalidation due to inactive timeout implies that
+ * no one is using the slot.
+ */
+ Assert(s->active_pid == 0);
+ }
+ break;
case RS_INVAL_NONE:
pg_unreachable();
}
@@ -1676,11 +1759,13 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
active_pid = s->active_pid;
/*
- * If the slot can be acquired, do so and mark it invalidated
- * immediately. Otherwise we'll signal the owning process, below, and
- * retry.
+ * If the slot can be acquired, do so and mark it as invalidated. If
+ * the slot is already ours, mark it as invalidated. Otherwise, we'll
+ * signal the owning process below and retry.
*/
- if (active_pid == 0)
+ if (active_pid == 0 ||
+ (MyReplicationSlot == s &&
+ active_pid == MyProcPid))
{
MyReplicationSlot = s;
s->active_pid = MyProcPid;
@@ -1735,7 +1820,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
{
ReportSlotInvalidation(invalidation_cause, true, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
if (MyBackendType == B_STARTUP)
(void) SendProcSignal(active_pid,
@@ -1781,7 +1867,8 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
ReportSlotInvalidation(invalidation_cause, false, active_pid,
slotname, restart_lsn,
- oldestLSN, snapshotConflictHorizon);
+ oldestLSN, snapshotConflictHorizon,
+ inactive_since);
/* done with this slot for now */
break;
@@ -1804,6 +1891,7 @@ InvalidatePossiblyObsoleteSlot(ReplicationSlotInvalidationCause cause,
* - RS_INVAL_HORIZON: requires a snapshot <= the given horizon in the given
* db; dboid may be InvalidOid for shared relations
* - RS_INVAL_WAL_LEVEL: is logical
+ * - RS_INVAL_INACTIVE_TIMEOUT: inactive slot timeout has occurred
*
* NB - this runs as part of checkpoint, so avoid raising errors if possible.
*/
@@ -1856,7 +1944,8 @@ restart:
}
/*
- * Flush all replication slots to disk.
+ * Flush all replication slots to disk. Also, invalidate obsolete slots during
+ * non-shutdown checkpoint.
*
* It is convenient to flush dirty replication slots at the time of checkpoint.
* Additionally, in case of a shutdown checkpoint, we also identify the slots
@@ -1914,6 +2003,38 @@ CheckPointReplicationSlots(bool is_shutdown)
SaveSlotToPath(s, path, LOG);
}
LWLockRelease(ReplicationSlotAllocationLock);
+
+ if (!is_shutdown)
+ {
+ elog(DEBUG1, "performing replication slot invalidation checks");
+
+ /*
+ * NB: We will make another pass over replication slots for
+ * invalidation checks to keep the code simple. Testing shows that
+ * there is no noticeable overhead (when compared with wal_removed
+ * invalidation) even if we were to do inactive_timeout invalidation
+ * of thousands of replication slots here. If it is ever proven that
+ * this assumption is wrong, we will have to perform the invalidation
+ * checks in the above for loop with the following changes:
+ *
+ * - Acquire ControlLock lock once before the loop.
+ *
+ * - Call InvalidatePossiblyObsoleteSlot for each slot.
+ *
+ * - Handle the cases in which ControlLock gets released just like
+ * InvalidateObsoleteReplicationSlots does.
+ *
+ * - Avoid saving slot info to disk two times for each invalidated
+ * slot.
+ *
+ * XXX: Should we move inactive_timeout invalidation check closer to
+ * wal_removed in CreateCheckPoint and CreateRestartPoint?
+ */
+ InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
+ 0,
+ InvalidOid,
+ InvalidTransactionId);
+ }
}
/*
@@ -2208,6 +2329,7 @@ RestoreSlotFromDisk(const char *name)
bool restored = false;
int readBytes;
pg_crc32c checksum;
+ TimestampTz now;
/* no need to lock here, no concurrent access allowed yet */
@@ -2368,6 +2490,9 @@ RestoreSlotFromDisk(const char *name)
NameStr(cp.slotdata.name)),
errhint("Change \"wal_level\" to be \"replica\" or higher.")));
+ /* Use same inactive_since time for all slots */
+ now = GetCurrentTimestamp();
+
/* nothing can be active yet, don't lock anything */
for (i = 0; i < max_replication_slots; i++)
{
@@ -2400,7 +2525,7 @@ RestoreSlotFromDisk(const char *name)
* slot from the disk into memory. Whoever acquires the slot i.e.
* makes the slot active will reset it.
*/
- slot->inactive_since = GetCurrentTimestamp();
+ ReplicationSlotSetInactiveSince(slot, &now, false);
restored = true;
break;
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index 488a161b3e..578cff64c8 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -536,7 +536,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS)
moveto = Min(moveto, GetXLogReplayRecPtr(NULL));
/* Acquire the slot so we "own" it */
- ReplicationSlotAcquire(NameStr(*slotname), true);
+ ReplicationSlotAcquire(NameStr(*slotname), true, true);
/* A slot whose restart_lsn has never been reserved cannot be advanced */
if (XLogRecPtrIsInvalid(MyReplicationSlot->data.restart_lsn))
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 371eef3ddd..b36ae90b2c 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -816,7 +816,7 @@ StartReplication(StartReplicationCmd *cmd)
if (cmd->slotname)
{
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
if (SlotIsLogical(MyReplicationSlot))
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
@@ -1434,7 +1434,7 @@ StartLogicalReplication(StartReplicationCmd *cmd)
Assert(!MyReplicationSlot);
- ReplicationSlotAcquire(cmd->slotname, true);
+ ReplicationSlotAcquire(cmd->slotname, true, true);
/*
* Force a disconnect, so that the decoding code doesn't need to care
diff --git a/src/backend/utils/adt/pg_upgrade_support.c b/src/backend/utils/adt/pg_upgrade_support.c
index 8a45b5827e..3322848e03 100644
--- a/src/backend/utils/adt/pg_upgrade_support.c
+++ b/src/backend/utils/adt/pg_upgrade_support.c
@@ -298,7 +298,7 @@ binary_upgrade_logical_slot_has_caught_up(PG_FUNCTION_ARGS)
slot_name = PG_GETARG_NAME(0);
/* Acquire the given slot */
- ReplicationSlotAcquire(NameStr(*slot_name), true);
+ ReplicationSlotAcquire(NameStr(*slot_name), true, false);
Assert(SlotIsLogical(MyReplicationSlot));
diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c
index 8a67f01200..367f510118 100644
--- a/src/backend/utils/misc/guc_tables.c
+++ b/src/backend/utils/misc/guc_tables.c
@@ -3028,6 +3028,18 @@ struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"replication_slot_inactive_timeout", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the amount of time a replication slot can remain inactive before "
+ "it will be invalidated."),
+ NULL,
+ GUC_UNIT_S
+ },
+ &replication_slot_inactive_timeout,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
{
{"commit_delay", PGC_SUSET, WAL_SETTINGS,
gettext_noop("Sets the delay in microseconds between transaction commit and "
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 39a3ac2312..7c6ae1baa2 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -336,6 +336,7 @@
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#track_commit_timestamp = off # collect timestamp of transaction commit
# (change requires restart)
+#replication_slot_inactive_timeout = 0 # in seconds; 0 disables
# - Primary Server -
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 45582cf9d8..8ffd06dd9f 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -56,6 +56,8 @@ typedef enum ReplicationSlotInvalidationCause
RS_INVAL_HORIZON,
/* wal_level insufficient for slot */
RS_INVAL_WAL_LEVEL,
+ /* inactive slot timeout has occurred */
+ RS_INVAL_INACTIVE_TIMEOUT,
} ReplicationSlotInvalidationCause;
extern PGDLLIMPORT const char *const SlotInvalidationCauses[];
@@ -224,6 +226,25 @@ typedef struct ReplicationSlotCtlData
ReplicationSlot replication_slots[1];
} ReplicationSlotCtlData;
+/*
+ * Set slot's inactive_since property unless it was previously invalidated.
+ */
+static inline void
+ReplicationSlotSetInactiveSince(ReplicationSlot *s, TimestampTz *now,
+ bool acquire_lock)
+{
+ if (s->data.invalidated != RS_INVAL_NONE)
+ return;
+
+ if (acquire_lock)
+ SpinLockAcquire(&s->mutex);
+
+ s->inactive_since = *now;
+
+ if (acquire_lock)
+ SpinLockRelease(&s->mutex);
+}
+
/*
* Pointers to shared memory
*/
@@ -233,6 +254,7 @@ extern PGDLLIMPORT ReplicationSlot *MyReplicationSlot;
/* GUCs */
extern PGDLLIMPORT int max_replication_slots;
extern PGDLLIMPORT char *synchronized_standby_slots;
+extern PGDLLIMPORT int replication_slot_inactive_timeout;
/* shmem initialization functions */
extern Size ReplicationSlotsShmemSize(void);
@@ -249,7 +271,8 @@ extern void ReplicationSlotDropAcquired(void);
extern void ReplicationSlotAlter(const char *name, const bool *failover,
const bool *two_phase);
-extern void ReplicationSlotAcquire(const char *name, bool nowait);
+extern void ReplicationSlotAcquire(const char *name, bool nowait,
+ bool error_if_invalid);
extern void ReplicationSlotRelease(void);
extern void ReplicationSlotCleanup(bool synced_only);
extern void ReplicationSlotSave(void);
diff --git a/src/test/recovery/meson.build b/src/test/recovery/meson.build
index b1eb77b1ec..708a2a3798 100644
--- a/src/test/recovery/meson.build
+++ b/src/test/recovery/meson.build
@@ -51,6 +51,7 @@ tests += {
't/040_standby_failover_slots_sync.pl',
't/041_checkpoint_at_promote.pl',
't/042_low_level_backup.pl',
+ 't/050_invalidate_slots.pl',
],
},
}
diff --git a/src/test/recovery/t/050_invalidate_slots.pl b/src/test/recovery/t/050_invalidate_slots.pl
new file mode 100644
index 0000000000..270f87c10a
--- /dev/null
+++ b/src/test/recovery/t/050_invalidate_slots.pl
@@ -0,0 +1,267 @@
+# Copyright (c) 2024, PostgreSQL Global Development Group
+
+# Test for replication slots invalidation
+use strict;
+use warnings FATAL => 'all';
+
+use PostgreSQL::Test::Utils;
+use PostgreSQL::Test::Cluster;
+use Test::More;
+use Time::HiRes qw(usleep);
+
+# =============================================================================
+# Testcase start
+#
+# Test invalidation of streaming standby slot and logical failover slot on the
+# primary due to inactive timeout. Also, test logical failover slot synced to
+# the standby from the primary doesn't get invalidated on its own, but gets the
+# invalidated state from the primary.
+
+# Initialize primary
+my $primary = PostgreSQL::Test::Cluster->new('primary');
+$primary->init(allows_streaming => 'logical');
+
+# Avoid unpredictability
+$primary->append_conf(
+ 'postgresql.conf', qq{
+checkpoint_timeout = 1h
+autovacuum = off
+});
+$primary->start;
+
+# Take backup
+my $backup_name = 'my_backup';
+$primary->backup($backup_name);
+
+# Create standby
+my $standby1 = PostgreSQL::Test::Cluster->new('standby1');
+$standby1->init_from_backup($primary, $backup_name, has_streaming => 1);
+
+my $connstr = $primary->connstr;
+$standby1->append_conf(
+ 'postgresql.conf', qq(
+hot_standby_feedback = on
+primary_slot_name = 'sb_slot1'
+primary_conninfo = '$connstr dbname=postgres'
+));
+
+# Create sync slot on the primary
+$primary->psql('postgres',
+ q{SELECT pg_create_logical_replication_slot('sync_slot1', 'test_decoding', false, false, true);}
+);
+
+# Create standby slot on the primary
+$primary->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_physical_replication_slot(slot_name := 'sb_slot1', immediately_reserve := true);
+]);
+
+$standby1->start;
+
+# Wait until the standby has replayed enough data
+$primary->wait_for_catchup($standby1);
+
+my $logstart = -s $standby1->logfile;
+
+# Set timeout GUC on the standby to verify that the next checkpoint will not
+# invalidate synced slots.
+my $inactive_timeout = 1;
+$standby1->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$standby1->reload;
+
+# Sync the primary slots to the standby
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+
+# Confirm that the logical failover slot is created on the standby
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND synced
+ AND NOT temporary
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'logical slot sync_slot1 is synced to standby');
+
+# Give enough time
+sleep($inactive_timeout + 1);
+
+# Despite inactive timeout being set, the synced slot won't get invalidated on
+# its own on the standby. So, we must not see invalidation message in server
+# log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+is( $standby1->safe_psql(
+ 'postgres',
+ q{SELECT count(*) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1'
+ AND invalidation_reason IS NULL;}
+ ),
+ "t",
+ 'check that synced slot sync_slot1 has not been invalidated on standby');
+
+$logstart = -s $primary->logfile;
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$primary->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '${inactive_timeout}s';
+]);
+$primary->reload;
+
+# Wait for logical failover slot to become inactive on the primary. Note that
+# nobody has acquired the slot yet, so it must get invalidated due to
+# inactive timeout.
+wait_for_slot_invalidation($primary, 'sync_slot1', $logstart,
+ $inactive_timeout);
+
+# Re-sync the primary slots to the standby. Note that the primary slot was
+# already invalidated (above) due to inactive timeout. The standby must just
+# sync the invalidated state.
+$standby1->safe_psql('postgres', "SELECT pg_sync_replication_slots();");
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'sync_slot1' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for sync_slot1 invalidation to be synced on standby";
+
+# Make the standby slot on the primary inactive and check for invalidation
+$standby1->stop;
+wait_for_slot_invalidation($primary, 'sb_slot1', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# =============================================================================
+# Testcase start
+# Invalidate logical subscriber slot due to inactive timeout.
+
+my $publisher = $primary;
+
+# Prepare for test
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO '0';
+]);
+$publisher->reload;
+
+# Create subscriber
+my $subscriber = PostgreSQL::Test::Cluster->new('sub');
+$subscriber->init;
+$subscriber->start;
+
+# Create tables
+$publisher->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+$subscriber->safe_psql('postgres', "CREATE TABLE test_tbl (id int)");
+
+# Insert some data
+$publisher->safe_psql('postgres',
+ "INSERT INTO test_tbl VALUES (generate_series(1, 5));");
+
+# Setup logical replication
+my $publisher_connstr = $publisher->connstr . ' dbname=postgres';
+$publisher->safe_psql('postgres', "CREATE PUBLICATION pub FOR ALL TABLES");
+$publisher->safe_psql(
+ 'postgres', qq[
+ SELECT pg_create_logical_replication_slot(slot_name := 'lsub1_slot', plugin := 'pgoutput');
+]);
+
+$subscriber->safe_psql('postgres',
+ "CREATE SUBSCRIPTION sub CONNECTION '$publisher_connstr' PUBLICATION pub WITH (slot_name = 'lsub1_slot', create_slot = false)"
+);
+$subscriber->wait_for_subscription_sync($publisher, 'sub');
+my $result =
+ $subscriber->safe_psql('postgres', "SELECT count(*) FROM test_tbl");
+is($result, qq(5), "check initial copy was done");
+
+# Set timeout GUC so that the next checkpoint will invalidate inactive slots
+$publisher->safe_psql(
+ 'postgres', qq[
+ ALTER SYSTEM SET replication_slot_inactive_timeout TO ' ${inactive_timeout}s';
+]);
+$publisher->reload;
+
+$logstart = -s $publisher->logfile;
+
+# Make subscriber slot on publisher inactive and check for invalidation
+$subscriber->stop;
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
+
+# Testcase end
+# =============================================================================
+
+# Wait for slot to first become inactive and then get invalidated
+sub wait_for_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+
+ # Wait for slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND active = 'f' AND
+ inactive_since IS NOT NULL;
+ ])
+ or die
+ "Timed out while waiting for slot $slot to become inactive on node $node_name";
+
+ trigger_slot_invalidation($node, $slot, $offset, $inactive_timeout);
+
+ # Check that an invalidated slot cannot be acquired
+ my ($result, $stdout, $stderr);
+ ($result, $stdout, $stderr) = $node->psql(
+ 'postgres', qq[
+ SELECT pg_replication_slot_advance('$slot', '0/1');
+ ]);
+ ok( $stderr =~ /can no longer get changes from replication slot "$slot"/,
+ "detected error upon trying to acquire invalidated slot $slot on node $node_name"
+ )
+ or die
+ "could not detect error upon trying to acquire invalidated slot $slot on node $node_name";
+}
+
+# Trigger slot invalidation and confirm it in the server log
+sub trigger_slot_invalidation
+{
+ my ($node, $slot, $offset, $inactive_timeout) = @_;
+ my $node_name = $node->name;
+ my $invalidated = 0;
+
+ # Give enough time to avoid multiple checkpoints
+ sleep($inactive_timeout + 1);
+
+ for (my $i = 0; $i < 10 * $PostgreSQL::Test::Utils::timeout_default; $i++)
+ {
+ $node->safe_psql('postgres', "CHECKPOINT");
+ if ($node->log_contains(
+ "invalidating obsolete replication slot \"$slot\"", $offset))
+ {
+ $invalidated = 1;
+ last;
+ }
+ usleep(100_000);
+ }
+ ok($invalidated,
+ "check that slot $slot invalidation has been logged on node $node_name"
+ );
+
+ # Check that the invalidation reason is 'inactive_timeout'
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for invalidation reason of slot $slot to be set on node $node_name";
+}
+
+done_testing();
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-08 11:54 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-09-16 10:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-11-07 10:03 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Nisha Moond <[email protected]>
@ 2024-11-13 09:29 ` vignesh C <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: vignesh C @ 2024-11-13 09:29 UTC (permalink / raw)
To: Nisha Moond <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, 7 Nov 2024 at 15:33, Nisha Moond <[email protected]> wrote:
>
> On Mon, Sep 16, 2024 at 3:31 PM Bharath Rupireddy
> <[email protected]> wrote:
> >
> > Please find the attached v46 patch having changes for the above review
> > comments and your test review comments and Shveta's review comments.
> >
> Hi,
>
> I’ve reviewed this thread and am interested in working on the
> remaining tasks and comments, as well as the future review comments.
> However, Bharath, please let me know if you'd prefer to continue with
> it.
>
> Attached the rebased v47 patch, which also addresses Peter’s comments
> #2, #3, and #4 at [1]. I will try addressing other comments as well in
> next versions.
The following crash occurs while upgrading:
2024-11-13 14:19:45.955 IST [44539] LOG: checkpoint starting: time
TRAP: failed Assert("!(*invalidated && SlotIsLogical(s) &&
IsBinaryUpgrade)"), File: "slot.c", Line: 1793, PID: 44539
postgres: checkpointer (ExceptionalCondition+0xbb)[0x555555e305bd]
postgres: checkpointer (+0x63ab04)[0x555555b8eb04]
postgres: checkpointer
(InvalidateObsoleteReplicationSlots+0x149)[0x555555b8ee5f]
postgres: checkpointer (CheckPointReplicationSlots+0x267)[0x555555b8f125]
postgres: checkpointer (+0x1f3ee8)[0x555555747ee8]
postgres: checkpointer (CreateCheckPoint+0x78f)[0x5555557475ee]
postgres: checkpointer (CheckpointerMain+0x632)[0x555555b2f1e7]
postgres: checkpointer (postmaster_child_launch+0x119)[0x555555b30892]
postgres: checkpointer (+0x5e2dc8)[0x555555b36dc8]
postgres: checkpointer (PostmasterMain+0x14bd)[0x555555b33647]
postgres: checkpointer (+0x487f2e)[0x5555559dbf2e]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7ffff6c29d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7ffff6c29e40]
postgres: checkpointer (_start+0x25)[0x555555634c25]
2024-11-13 14:19:45.967 IST [44538] LOG: checkpointer process (PID
44539) was terminated by signal 6: Aborted
This can happen in the following case:
1) Setup a logical replication cluster with enough data so that it
will take at least few minutes to upgrade
2) Stop the publisher node
3) Configure replication_slot_inactive_timeout and checkpoint_timeout
to 30 seconds
4) Upgrade the publisher node.
This is happening because logical replication slots are getting
invalidated during upgrade and there is an assertion which checks that
the slots are not invalidated.
I feel this can be fixed by having a function similar to
check_max_slot_wal_keep_size which will make sure that
replication_slot_inactive_timeout is 0 during upgrade.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-02 09:50 ` Amit Kapila <[email protected]>
3 siblings, 0 replies; 98+ messages in thread
From: Amit Kapila @ 2024-09-02 09:50 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Sat, Aug 31, 2024 at 1:45 PM Bharath Rupireddy
<[email protected]> wrote:
>
> Please find the attached v44 patch with the above changes. I will
> include the 0002 xid_age based invalidation patch later.
>
It is better to get the 0001 reviewed and committed first. We can
discuss about 0002 afterwards as 0001 is in itself a complete and
separate patch that can be committed.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-03 06:55 ` Peter Smith <[email protected]>
3 siblings, 0 replies; 98+ messages in thread
From: Peter Smith @ 2024-09-03 06:55 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi, my previous review posts did not cover the test code.
Here are my review comments for the v44-0001 test code
======
TEST CASE #1
1.
+# Wait for the inactive replication slot to be invalidated.
+$standby1->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = 'lsub1_sync_slot' AND
+ invalidation_reason = 'inactive_timeout';
+])
+ or die
+ "Timed out while waiting for lsub1_sync_slot invalidation to be
synced on standby";
+
Is that comment correct? IIUC the synced slot should *already* be
invalidated from the primary, so here we are not really "waiting" for
it to be invalidated; Instead, we are just "confirming" that the
synchronized slot is already invalidated with the correct reason as
expected.
~~~
2.
+# Synced slot mustn't get invalidated on the standby even after a checkpoint,
+# it must sync invalidation from the primary. So, we must not see the slot's
+# invalidation message in server log.
+$standby1->safe_psql('postgres', "CHECKPOINT");
+ok( !$standby1->log_contains(
+ "invalidating obsolete replication slot \"lsub1_sync_slot\"",
+ $standby1_logstart),
+ 'check that syned lsub1_sync_slot has not been invalidated on the standby'
+);
+
This test case seemed bogus, for a couple of reasons:
2a. IIUC this 'lsub1_sync_slot' is the same one that is already
invalid (from the primary), so nobody should be surprised that an
already invalid slot doesn't get flagged as invalid again. i.e.
Shouldn't your test scenario here be done using a valid synced slot?
2b. AFAICT it was only moments above this CHECKPOINT where you
assigned the standby inactivity timeout to 2s. So even if there was
some bug invalidating synced slots I don't think you gave it enough
time to happen -- e.g. I doubt 2s has elapsed yet.
~
3.
+# Stop standby to make the standby's replication slot on the primary inactive
+$standby1->stop;
+
+# Wait for the standby's replication slot to become inactive
+wait_for_slot_invalidation($primary, 'sb1_slot', $logstart,
+ $inactive_timeout);
This seems a bit tricky. Both these (the stop and the wait) seem to
belong together, so I think maybe a single bigger explanatory comment
covering both parts would help for understanding.
======
TEST CASE #2
4.
+# Stop subscriber to make the replication slot on publisher inactive
+$subscriber->stop;
+
+# Wait for the replication slot to become inactive and then invalidated due to
+# timeout.
+wait_for_slot_invalidation($publisher, 'lsub1_slot', $logstart,
+ $inactive_timeout);
IIUC, this is just like comment #3 above. Both these (the stop and the
wait) seem to belong together, so I think maybe a single bigger
explanatory comment covering both parts would help for understanding.
~~~
5.
+# Testcase end: Invalidate logical subscriber's slot due to
+# replication_slot_inactive_timeout.
+# =============================================================================
IMO the rest of the comment after "Testcase end" isn't very useful.
======
sub wait_for_slot_invalidation
6.
+sub wait_for_slot_invalidation
+{
An explanatory header comment for this subroutine would be helpful.
~~~
7.
+ # Wait for the replication slot to become inactive
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for slot $slot_name to become inactive on
node $name";
+
+ # Wait for the replication slot info to be updated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE inactive_since IS NOT NULL
+ AND slot_name = '$slot_name' AND active = 'f';
+ ])
+ or die
+ "Timed out while waiting for info of slot $slot_name to be updated
on node $name";
+
Why are there are 2 separate poll_query_until's here? Can't those be
combined into just one?
~~~
8.
+ # Sleep at least $inactive_timeout duration to avoid multiple checkpoints
+ # for the slot to get invalidated.
+ sleep($inactive_timeout);
+
Maybe this special sleep to prevent too many CHECKPOINTs should be
moved to be inside the other subroutine, which is actually doing those
CHECKPOINTs.
~~~
9.
+ # Wait for the inactive replication slot to be invalidated
+ $node->poll_query_until(
+ 'postgres', qq[
+ SELECT COUNT(slot_name) = 1 FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND
+ invalidation_reason = 'inactive_timeout';
+ ])
+ or die
+ "Timed out while waiting for inactive slot $slot_name to be
invalidated on node $name";
+
The comment seems misleading. IIUC you are not "waiting" for the
invalidation here, because it is the other subroutine doing the
waiting for the invalidation message in the logs. Instead, here I
think you are just confirming the 'invalidation_reason' got set
correctly. The comment should say what it is really doing.
======
sub check_for_slot_invalidation_in_server_log
10.
+# Check for invalidation of slot in server log
+sub check_for_slot_invalidation_in_server_log
+{
I think the main function of this subroutine is the CHECKPOINT and the
waiting for the server log to say invalidation happened. It is doing a
loop of a) CHECKPOINT then b) inspecting the server log for the slot
invalidation, and c) waiting for a bit. Repeat 10 times.
A comment describing the logic for this subroutine would be helpful.
The most important side-effect of this function is the CHECKPOINT
because without that nothing will ever get invalidated due to
inactivity, but this key point is not obvious from the subroutine
name.
IMO it would be better to name this differently to reflect what it is
really doing:
e.g. "CHECKPOINT_and_wait_for_slot_invalidation_in_server_log"
======
Kind Regards,
Peter Smith.
Fujitsu Australia
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-03 09:31 ` shveta malik <[email protected]>
2024-09-04 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-08 11:55 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
3 siblings, 2 replies; 98+ messages in thread
From: shveta malik @ 2024-09-03 09:31 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Sat, Aug 31, 2024 at 1:45 PM Bharath Rupireddy
<[email protected]> wrote:
>
> Hi,
>
>
> Please find the attached v44 patch with the above changes. I will
> include the 0002 xid_age based invalidation patch later.
>
Thanks for the patch Bharath. My review and testing is WIP, but please
find few comments and queries:
1)
I see that ReplicationSlotAlter() will error out if the slot is
invalidated due to timeout. I have not tested it myself, but do you
know if slot-alter errors out for other invalidation causes as well?
Just wanted to confirm that the behaviour is consistent for all
invalidation causes.
2)
When a slot is invalidated, and we try to use that slot, it gives this msg:
ERROR: can no longer get changes from replication slot "mysubnew1_2"
DETAIL: The slot became invalid because it was inactive since
2024-09-03 14:23:34.094067+05:30, which is more than 600 seconds ago.
HINT: You might need to increase "replication_slot_inactive_timeout.".
Isn't HINT misleading? Even if we increase it now, the slot can not be
reused again.
3)
When the slot is invalidated, the' inactive_since' still keeps on
changing when there is a subscriber trying to start replication
continuously. I think ReplicationSlotAcquire() keeps on failing and
thus Release keeps on setting it again and again. Shouldn't we stop
setting/chnaging 'inactive_since' once the slot is invalidated
already, otherwise it will be misleading.
postgres=# select failover,synced,inactive_since,invalidation_reason
from pg_replication_slots;
failover | synced | inactive_since | invalidation_reason
----------+--------+----------------------------------+---------------------
t | f | 2024-09-03 14:23:.. | inactive_timeout
after sometime:
failover | synced | inactive_since | invalidation_reason
----------+--------+----------------------------------+---------------------
t | f | 2024-09-03 14:26:..| inactive_timeout
4)
src/sgml/config.sgml:
4a)
+ A value of zero (which is default) disables the timeout mechanism.
Better will be:
A value of zero (which is default) disables the inactive timeout
invalidation mechanism .
or
A value of zero (which is default) disables the slot invalidation due
to the inactive timeout mechanism.
i.e. rephrase to indicate that invalidation is disabled.
4b)
'synced' and inactive_since should point to pg_replication_slots:
example:
<link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
5)
src/sgml/system-views.sgml:
+ ..the slot has been inactive for longer than the duration specified
by replication_slot_inactive_timeout parameter.
Better to have:
..the slot has been inactive for a time longer than the duration
specified by the replication_slot_inactive_timeout parameter.
thanks
Shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-04 03:47 ` shveta malik <[email protected]>
2024-09-04 09:18 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-05 04:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
1 sibling, 2 replies; 98+ messages in thread
From: shveta malik @ 2024-09-04 03:47 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Tue, Sep 3, 2024 at 3:01 PM shveta malik <[email protected]> wrote:
>
>
> 1)
> I see that ReplicationSlotAlter() will error out if the slot is
> invalidated due to timeout. I have not tested it myself, but do you
> know if slot-alter errors out for other invalidation causes as well?
> Just wanted to confirm that the behaviour is consistent for all
> invalidation causes.
I was able to test this and as anticipated behavior is different. When
slot is invalidated due to say 'wal_removed', I am still able to do
'alter' of that slot.
Please see:
Pub:
slot_name | failover | synced | inactive_since |
invalidation_reason
-------------+----------+--------+----------------------------------+---------------------
mysubnew1_1 | t | f | 2024-09-04 08:58:12.802278+05:30 |
wal_removed
Sub:
newdb1=# alter subscription mysubnew1_1 disable;
ALTER SUBSCRIPTION
newdb1=# alter subscription mysubnew1_1 set (failover=false);
ALTER SUBSCRIPTION
Pub: (failover altered)
slot_name | failover | synced | inactive_since |
invalidation_reason
-------------+----------+--------+----------------------------------+---------------------
mysubnew1_1 | f | f | 2024-09-04 08:58:47.824471+05:30 |
wal_removed
while when invalidation_reason is 'inactive_timeout', it fails:
Pub:
slot_name | failover | synced | inactive_since |
invalidation_reason
-------------+----------+--------+----------------------------------+---------------------
mysubnew1_1 | t | f | 2024-09-03 14:30:57.532206+05:30 |
inactive_timeout
Sub:
newdb1=# alter subscription mysubnew1_1 disable;
ALTER SUBSCRIPTION
newdb1=# alter subscription mysubnew1_1 set (failover=false);
ERROR: could not alter replication slot "mysubnew1_1": ERROR: can no
longer get changes from replication slot "mysubnew1_1"
DETAIL: The slot became invalid because it was inactive since
2024-09-04 08:54:20.308996+05:30, which is more than 0 seconds ago.
HINT: You might need to increase "replication_slot_inactive_timeout.".
I think the behavior should be same.
thanks
Shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-04 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-04 09:18 ` shveta malik <[email protected]>
2024-09-05 03:55 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: shveta malik @ 2024-09-04 09:18 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Wed, Sep 4, 2024 at 9:17 AM shveta malik <[email protected]> wrote:
>
> On Tue, Sep 3, 2024 at 3:01 PM shveta malik <[email protected]> wrote:
> >
> >
1)
It is related to one of my previous comments (pt 3 in [1]) where I
stated that inactive_since should not keep on changing once a slot is
invalidated.
Below is one side effect if inactive_since keeps on changing:
postgres=# SELECT * FROM pg_replication_slot_advance('mysubnew1_1',
pg_current_wal_lsn());
ERROR: can no longer get changes from replication slot "mysubnew1_1"
DETAIL: The slot became invalid because it was inactive since
2024-09-04 10:03:56.68053+05:30, which is more than 10 seconds ago.
HINT: You might need to increase "replication_slot_inactive_timeout.".
postgres=# select now();
now
---------------------------------
2024-09-04 10:04:00.26564+05:30
'DETAIL' gives wrong information, we are not past 10-seconds. This is
because inactive_since got updated even in ERROR scenario.
2)
One more issue in this message is, once I set
replication_slot_inactive_timeout to a bigger value, it becomes more
misleading. This is because invalidation was done in the past using
previous value while message starts showing new value:
ALTER SYSTEM SET replication_slot_inactive_timeout TO '36h';
--see 129600 secs in DETAIL and the current time.
postgres=# SELECT * FROM pg_replication_slot_advance('mysubnew1_1',
pg_current_wal_lsn());
ERROR: can no longer get changes from replication slot "mysubnew1_1"
DETAIL: The slot became invalid because it was inactive since
2024-09-04 10:06:38.980939+05:30, which is more than 129600 seconds
ago.
postgres=# select now();
now
----------------------------------
2024-09-04 10:07:35.201894+05:30
I feel we should change this message itself.
~~~~~
When invalidation is due to wal_removed, we get a way simpler message:
newdb1=# SELECT * FROM pg_replication_slot_advance('mysubnew1_2',
pg_current_wal_lsn());
ERROR: replication slot "mysubnew1_2" cannot be advanced
DETAIL: This slot has never previously reserved WAL, or it has been
invalidated.
This message does not mention 'max_slot_wal_keep_size'. We should have
a similar message for our case. Thoughts?
[1]: https://www.postgresql.org/message-id/CAJpy0uC8Dg-0JS3NRUwVUemgz5Ar2v3_EQQFXyAigWSEQ8U47Q%40mail.gma...
thanks
Shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-04 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-04 09:18 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-05 03:55 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Amit Kapila @ 2024-09-05 03:55 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Sep 4, 2024 at 2:49 PM shveta malik <[email protected]> wrote:
>
> On Wed, Sep 4, 2024 at 9:17 AM shveta malik <[email protected]> wrote:
> >
> > On Tue, Sep 3, 2024 at 3:01 PM shveta malik <[email protected]> wrote:
> > >
> > >
>
>
> 1)
> It is related to one of my previous comments (pt 3 in [1]) where I
> stated that inactive_since should not keep on changing once a slot is
> invalidated.
>
Agreed. Updating the inactive_since for a slot that is already invalid
is misleading.
>
>
> 2)
> One more issue in this message is, once I set
> replication_slot_inactive_timeout to a bigger value, it becomes more
> misleading. This is because invalidation was done in the past using
> previous value while message starts showing new value:
>
> ALTER SYSTEM SET replication_slot_inactive_timeout TO '36h';
>
> --see 129600 secs in DETAIL and the current time.
> postgres=# SELECT * FROM pg_replication_slot_advance('mysubnew1_1',
> pg_current_wal_lsn());
> ERROR: can no longer get changes from replication slot "mysubnew1_1"
> DETAIL: The slot became invalid because it was inactive since
> 2024-09-04 10:06:38.980939+05:30, which is more than 129600 seconds
> ago.
> postgres=# select now();
> now
> ----------------------------------
> 2024-09-04 10:07:35.201894+05:30
>
> I feel we should change this message itself.
>
+1.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-04 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-05 04:00 ` Amit Kapila <[email protected]>
2024-09-09 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: Amit Kapila @ 2024-09-05 04:00 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Sep 4, 2024 at 9:17 AM shveta malik <[email protected]> wrote:
>
> On Tue, Sep 3, 2024 at 3:01 PM shveta malik <[email protected]> wrote:
> >
> >
> > 1)
> > I see that ReplicationSlotAlter() will error out if the slot is
> > invalidated due to timeout. I have not tested it myself, but do you
> > know if slot-alter errors out for other invalidation causes as well?
> > Just wanted to confirm that the behaviour is consistent for all
> > invalidation causes.
>
> I was able to test this and as anticipated behavior is different. When
> slot is invalidated due to say 'wal_removed', I am still able to do
> 'alter' of that slot.
> Please see:
>
> Pub:
> slot_name | failover | synced | inactive_since |
> invalidation_reason
> -------------+----------+--------+----------------------------------+---------------------
> mysubnew1_1 | t | f | 2024-09-04 08:58:12.802278+05:30 |
> wal_removed
>
> Sub:
> newdb1=# alter subscription mysubnew1_1 disable;
> ALTER SUBSCRIPTION
>
> newdb1=# alter subscription mysubnew1_1 set (failover=false);
> ALTER SUBSCRIPTION
>
> Pub: (failover altered)
> slot_name | failover | synced | inactive_since |
> invalidation_reason
> -------------+----------+--------+----------------------------------+---------------------
> mysubnew1_1 | f | f | 2024-09-04 08:58:47.824471+05:30 |
> wal_removed
>
>
> while when invalidation_reason is 'inactive_timeout', it fails:
>
> Pub:
> slot_name | failover | synced | inactive_since |
> invalidation_reason
> -------------+----------+--------+----------------------------------+---------------------
> mysubnew1_1 | t | f | 2024-09-03 14:30:57.532206+05:30 |
> inactive_timeout
>
> Sub:
> newdb1=# alter subscription mysubnew1_1 disable;
> ALTER SUBSCRIPTION
>
> newdb1=# alter subscription mysubnew1_1 set (failover=false);
> ERROR: could not alter replication slot "mysubnew1_1": ERROR: can no
> longer get changes from replication slot "mysubnew1_1"
> DETAIL: The slot became invalid because it was inactive since
> 2024-09-04 08:54:20.308996+05:30, which is more than 0 seconds ago.
> HINT: You might need to increase "replication_slot_inactive_timeout.".
>
> I think the behavior should be same.
>
We should not allow the invalid replication slot to be altered
irrespective of the reason unless there is any benefit.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-04 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-05 04:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
@ 2024-09-09 03:47 ` shveta malik <[email protected]>
2024-09-09 04:56 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: shveta malik @ 2024-09-09 03:47 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Thu, Sep 5, 2024 at 9:30 AM Amit Kapila <[email protected]> wrote:
>
> On Wed, Sep 4, 2024 at 9:17 AM shveta malik <[email protected]> wrote:
> >
> > On Tue, Sep 3, 2024 at 3:01 PM shveta malik <[email protected]> wrote:
> > >
> > >
> > > 1)
> > > I see that ReplicationSlotAlter() will error out if the slot is
> > > invalidated due to timeout. I have not tested it myself, but do you
> > > know if slot-alter errors out for other invalidation causes as well?
> > > Just wanted to confirm that the behaviour is consistent for all
> > > invalidation causes.
> >
> > I was able to test this and as anticipated behavior is different. When
> > slot is invalidated due to say 'wal_removed', I am still able to do
> > 'alter' of that slot.
> > Please see:
> >
> > Pub:
> > slot_name | failover | synced | inactive_since |
> > invalidation_reason
> > -------------+----------+--------+----------------------------------+---------------------
> > mysubnew1_1 | t | f | 2024-09-04 08:58:12.802278+05:30 |
> > wal_removed
> >
> > Sub:
> > newdb1=# alter subscription mysubnew1_1 disable;
> > ALTER SUBSCRIPTION
> >
> > newdb1=# alter subscription mysubnew1_1 set (failover=false);
> > ALTER SUBSCRIPTION
> >
> > Pub: (failover altered)
> > slot_name | failover | synced | inactive_since |
> > invalidation_reason
> > -------------+----------+--------+----------------------------------+---------------------
> > mysubnew1_1 | f | f | 2024-09-04 08:58:47.824471+05:30 |
> > wal_removed
> >
> >
> > while when invalidation_reason is 'inactive_timeout', it fails:
> >
> > Pub:
> > slot_name | failover | synced | inactive_since |
> > invalidation_reason
> > -------------+----------+--------+----------------------------------+---------------------
> > mysubnew1_1 | t | f | 2024-09-03 14:30:57.532206+05:30 |
> > inactive_timeout
> >
> > Sub:
> > newdb1=# alter subscription mysubnew1_1 disable;
> > ALTER SUBSCRIPTION
> >
> > newdb1=# alter subscription mysubnew1_1 set (failover=false);
> > ERROR: could not alter replication slot "mysubnew1_1": ERROR: can no
> > longer get changes from replication slot "mysubnew1_1"
> > DETAIL: The slot became invalid because it was inactive since
> > 2024-09-04 08:54:20.308996+05:30, which is more than 0 seconds ago.
> > HINT: You might need to increase "replication_slot_inactive_timeout.".
> >
> > I think the behavior should be same.
> >
>
> We should not allow the invalid replication slot to be altered
> irrespective of the reason unless there is any benefit.
>
Okay, then I think we need to change the existing behaviour of the
other invalidation causes which still allow alter-slot.
thanks
Shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-04 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-05 04:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-09-09 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-09 04:56 ` Bharath Rupireddy <[email protected]>
2024-09-09 04:58 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-09-09 04:56 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On Mon, Sep 9, 2024 at 9:17 AM shveta malik <[email protected]> wrote:
>
> > We should not allow the invalid replication slot to be altered
> > irrespective of the reason unless there is any benefit.
>
> Okay, then I think we need to change the existing behaviour of the
> other invalidation causes which still allow alter-slot.
+1. Perhaps, track it in a separate thread?
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-04 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-05 04:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-09-09 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-09 04:56 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-09 04:58 ` shveta malik <[email protected]>
2024-09-09 09:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: shveta malik @ 2024-09-09 04:58 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Sep 9, 2024 at 10:26 AM Bharath Rupireddy
<[email protected]> wrote:
>
> Hi,
>
> On Mon, Sep 9, 2024 at 9:17 AM shveta malik <[email protected]> wrote:
> >
> > > We should not allow the invalid replication slot to be altered
> > > irrespective of the reason unless there is any benefit.
> >
> > Okay, then I think we need to change the existing behaviour of the
> > other invalidation causes which still allow alter-slot.
>
> +1. Perhaps, track it in a separate thread?
I think so. It does not come under the scope of this thread.
thanks
Shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-04 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-05 04:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-09-09 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-09 04:56 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 04:58 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-09 09:34 ` Amit Kapila <[email protected]>
2024-09-09 18:42 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Amit Kapila @ 2024-09-09 09:34 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Sep 9, 2024 at 10:28 AM shveta malik <[email protected]> wrote:
>
> On Mon, Sep 9, 2024 at 10:26 AM Bharath Rupireddy
> <[email protected]> wrote:
> >
> > Hi,
> >
> > On Mon, Sep 9, 2024 at 9:17 AM shveta malik <[email protected]> wrote:
> > >
> > > > We should not allow the invalid replication slot to be altered
> > > > irrespective of the reason unless there is any benefit.
> > >
> > > Okay, then I think we need to change the existing behaviour of the
> > > other invalidation causes which still allow alter-slot.
> >
> > +1. Perhaps, track it in a separate thread?
>
> I think so. It does not come under the scope of this thread.
>
It makes sense to me as well. But let's go ahead and get that sorted out first.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-04 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-05 04:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-09-09 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-09 04:56 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 04:58 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-09 09:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
@ 2024-09-09 18:42 ` Bharath Rupireddy <[email protected]>
2024-09-16 03:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-09-09 18:42 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On Mon, Sep 9, 2024 at 3:04 PM Amit Kapila <[email protected]> wrote:
>
> > > > > We should not allow the invalid replication slot to be altered
> > > > > irrespective of the reason unless there is any benefit.
> > > >
> > > > Okay, then I think we need to change the existing behaviour of the
> > > > other invalidation causes which still allow alter-slot.
> > >
> > > +1. Perhaps, track it in a separate thread?
> >
> > I think so. It does not come under the scope of this thread.
>
> It makes sense to me as well. But let's go ahead and get that sorted out first.
Moved the discussion to new thread -
https://www.postgresql.org/message-id/CALj2ACW4fSOMiKjQ3%3D2NVBMTZRTG8Ujg6jsK9z3EvOtvA4vzKQ%40mail.g....
Please have a look.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-04 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-05 04:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-09-09 03:47 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-09 04:56 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-09 04:58 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-09-09 09:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-09-09 18:42 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-16 03:25 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Amit Kapila @ 2024-09-16 03:25 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: shveta malik <[email protected]>; Peter Smith <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Tue, Sep 10, 2024 at 12:13 AM Bharath Rupireddy
<[email protected]> wrote:
>
> On Mon, Sep 9, 2024 at 3:04 PM Amit Kapila <[email protected]> wrote:
> >
> > > > > > We should not allow the invalid replication slot to be altered
> > > > > > irrespective of the reason unless there is any benefit.
> > > > >
> > > > > Okay, then I think we need to change the existing behaviour of the
> > > > > other invalidation causes which still allow alter-slot.
> > > >
> > > > +1. Perhaps, track it in a separate thread?
> > >
> > > I think so. It does not come under the scope of this thread.
> >
> > It makes sense to me as well. But let's go ahead and get that sorted out first.
>
> Moved the discussion to new thread -
> https://www.postgresql.org/message-id/CALj2ACW4fSOMiKjQ3%3D2NVBMTZRTG8Ujg6jsK9z3EvOtvA4vzKQ%40mail.g....
> Please have a look.
>
That is pushed now. Please send the rebased patch after addressing the
pending comments.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Peter Smith <[email protected]>
2024-08-31 08:15 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-09-03 09:31 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
@ 2024-09-08 11:55 ` Bharath Rupireddy <[email protected]>
1 sibling, 0 replies; 98+ messages in thread
From: Bharath Rupireddy @ 2024-09-08 11:55 UTC (permalink / raw)
To: shveta malik <[email protected]>; +Cc: Peter Smith <[email protected]>; Amit Kapila <[email protected]>; Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
Thanks for reviewing.
On Tue, Sep 3, 2024 at 3:01 PM shveta malik <[email protected]> wrote:
>
> 1)
> I see that ReplicationSlotAlter() will error out if the slot is
> invalidated due to timeout. I have not tested it myself, but do you
> know if slot-alter errors out for other invalidation causes as well?
> Just wanted to confirm that the behaviour is consistent for all
> invalidation causes.
Will respond to Amit's comment soon.
> 2)
> When a slot is invalidated, and we try to use that slot, it gives this msg:
>
> ERROR: can no longer get changes from replication slot "mysubnew1_2"
> DETAIL: The slot became invalid because it was inactive since
> 2024-09-03 14:23:34.094067+05:30, which is more than 600 seconds ago.
> HINT: You might need to increase "replication_slot_inactive_timeout.".
>
> Isn't HINT misleading? Even if we increase it now, the slot can not be
> reused again.
>
> Below is one side effect if inactive_since keeps on changing:
>
> postgres=# SELECT * FROM pg_replication_slot_advance('mysubnew1_1',
> pg_current_wal_lsn());
> ERROR: can no longer get changes from replication slot "mysubnew1_1"
> DETAIL: The slot became invalid because it was inactive since
> 2024-09-04 10:03:56.68053+05:30, which is more than 10 seconds ago.
> HINT: You might need to increase "replication_slot_inactive_timeout.".
>
> postgres=# select now();
> now
> ---------------------------------
> 2024-09-04 10:04:00.26564+05:30
>
> 'DETAIL' gives wrong information, we are not past 10-seconds. This is
> because inactive_since got updated even in ERROR scenario.
>
> ERROR: can no longer get changes from replication slot "mysubnew1_1"
> DETAIL: The slot became invalid because it was inactive since
> 2024-09-04 10:06:38.980939+05:30, which is more than 129600 seconds
> ago.
> postgres=# select now();
> now
> ----------------------------------
> 2024-09-04 10:07:35.201894+05:30
>
> I feel we should change this message itself.
Removed the hint and corrected the detail message as following:
errmsg("can no longer get changes from replication slot \"%s\"",
NameStr(s->data.name)),
errdetail("This slot has been invalidated because it was inactive for
longer than the amount of time specified by \"%s\".",
"replication_slot_inactive_timeout.")));
> 3)
> When the slot is invalidated, the' inactive_since' still keeps on
> changing when there is a subscriber trying to start replication
> continuously. I think ReplicationSlotAcquire() keeps on failing and
> thus Release keeps on setting it again and again. Shouldn't we stop
> setting/chnaging 'inactive_since' once the slot is invalidated
> already, otherwise it will be misleading.
>
> postgres=# select failover,synced,inactive_since,invalidation_reason
> from pg_replication_slots;
>
> failover | synced | inactive_since | invalidation_reason
> ----------+--------+----------------------------------+---------------------
> t | f | 2024-09-03 14:23:.. | inactive_timeout
>
> after sometime:
> failover | synced | inactive_since | invalidation_reason
> ----------+--------+----------------------------------+---------------------
> t | f | 2024-09-03 14:26:..| inactive_timeout
Changed it to not update inactive_since for slots invalidated due to
inactive timeout.
> 4)
> src/sgml/config.sgml:
>
> 4a)
> + A value of zero (which is default) disables the timeout mechanism.
>
> Better will be:
> A value of zero (which is default) disables the inactive timeout
> invalidation mechanism .
Changed.
> 4b)
> 'synced' and inactive_since should point to pg_replication_slots:
>
> example:
> <link linkend="view-pg-replication-slots">pg_replication_slots</link>.<structfield>synced</structfield>
Modified.
> 5)
> src/sgml/system-views.sgml:
> + ..the slot has been inactive for longer than the duration specified
> by replication_slot_inactive_timeout parameter.
>
> Better to have:
> ..the slot has been inactive for a time longer than the duration
> specified by the replication_slot_inactive_timeout parameter.
Changed it to the following to be consistent with the config.sgml.
<literal>inactive_timeout</literal> means that the slot has been
inactive for longer than the amount of time specified by the
<xref linkend="guc-replication-slot-inactive-timeout"/> parameter.
Please find the v45 patch posted upthread at
https://www.postgresql.org/message-id/CALj2ACWXQT3_HY40ceqKf1DadjLQP6b1r%3D0sZRh-xhAOd-b0pA%40mail.g...
for the changes.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-06 11:40 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-06-24 06:00 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-14 03:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-08-29 06:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-09-02 06:55 ` Amit Kapila <[email protected]>
1 sibling, 0 replies; 98+ messages in thread
From: Amit Kapila @ 2024-09-02 06:55 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Ajin Cherian <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Aug 29, 2024 at 11:31 AM Bharath Rupireddy
<[email protected]> wrote:
>
> Thanks for looking into this.
>
> On Mon, Aug 26, 2024 at 4:35 PM Amit Kapila <[email protected]> wrote:
> >
> > Few comments on 0001:
> > 1.
> > @@ -651,6 +651,13 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid
> >
> > + /*
> > + * Skip the sync if the local slot is already invalidated. We do this
> > + * beforehand to avoid slot acquire and release.
> > + */
> >
> > I was wondering why you have added this new check as part of this
> > patch. If you see the following comments in the related code, you will
> > know why we haven't done this previously.
>
> Removed. Can deal with optimization separately.
>
> > 2.
> > + */
> > + InvalidateObsoleteReplicationSlots(RS_INVAL_INACTIVE_TIMEOUT,
> > + 0, InvalidOid, InvalidTransactionId);
> >
> > Why do we want to call this for shutdown case (when is_shutdown is
> > true)? I understand trying to invalidate slots during regular
> > checkpoint but not sure if we need it at the time of shutdown.
>
> Changed it to invalidate only for non-shutdown checkpoints. inactive_timeout invalidation isn't critical for shutdown unlike wal_removed which can help shutdown by freeing up some disk space.
>
> > The
> > other point is can we try to check the performance impact with 100s of
> > slots as mentioned in the code comments?
>
> I first checked how much does the wal_removed invalidation check add to the checkpoint (see 2nd and 3rd column). I then checked how much inactive_timeout invalidation check adds to the checkpoint (see 4th column), it is not more than wal_remove invalidation check. I then checked how much the wal_removed invalidation check adds for replication slots that have already been invalidated due to inactive_timeout (see 5th column), looks like not much.
>
> | # of slots | HEAD (no invalidation) ms | HEAD (wal_removed) ms | PATCHED (inactive_timeout) ms | PATCHED (inactive_timeout+wal_removed) ms |
> |------------|----------------------------|-----------------------|-------------------------------|------------------------------------------|
> | 100 | 18.591 | 370.586 | 359.299 | 373.882 |
> | 1000 | 15.722 | 4834.901 | 5081.751 | 5072.128 |
> | 10000 | 19.261 | 59801.062 | 61270.406 | 60270.099 |
>
> Having said that, I'm okay to implement the optimization specified. Thoughts?
>
The other possibility is to try invalidating due to timeout along with
wal_removed case during checkpoint. The idea is that if the slot can
be invalidated due to WAL then fine, otherwise check if it can be
invalidated due to timeout. This can avoid looping the slots and doing
similar work multiple times during the checkpoint.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-04 04:11 ` Masahiko Sawada <[email protected]>
2024-04-04 04:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2 siblings, 1 reply; 98+ messages in thread
From: Masahiko Sawada @ 2024-04-04 04:11 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Apr 3, 2024 at 11:58 PM Bharath Rupireddy
<[email protected]> wrote:
>
>
> Please find the attached v33 patches.
@@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
if (SlotSyncCtx->pid == InvalidPid)
{
SpinLockRelease(&SlotSyncCtx->mutex);
+ update_synced_slots_inactive_since();
return;
}
SpinLockRelease(&SlotSyncCtx->mutex);
@@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
}
SpinLockRelease(&SlotSyncCtx->mutex);
+
+ update_synced_slots_inactive_since();
}
Why do we want to update all synced slots' inactive_since values at
shutdown in spite of updating the value every time when releasing the
slot? It seems to contradict the fact that inactive_since is updated
when releasing or restoring the slot.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 04:11 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
@ 2024-04-04 04:34 ` Bharath Rupireddy <[email protected]>
2024-04-04 08:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-04-04 04:34 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada <[email protected]> wrote:
>
> @@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
> if (SlotSyncCtx->pid == InvalidPid)
> {
> SpinLockRelease(&SlotSyncCtx->mutex);
> + update_synced_slots_inactive_since();
> return;
> }
> SpinLockRelease(&SlotSyncCtx->mutex);
> @@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
> }
>
> SpinLockRelease(&SlotSyncCtx->mutex);
> +
> + update_synced_slots_inactive_since();
> }
>
> Why do we want to update all synced slots' inactive_since values at
> shutdown in spite of updating the value every time when releasing the
> slot? It seems to contradict the fact that inactive_since is updated
> when releasing or restoring the slot.
It is to get the inactive_since right for the cases where the standby
is promoted without a restart similar to when a standby is promoted
with restart in which case the inactive_since is set to current time
in RestoreSlotFromDisk.
Imagine the slot is synced last time at time t1 and then a few hours
passed, the standby is promoted without a restart. If we don't set
inactive_since to current time in this case in ShutDownSlotSync, the
inactive timeout invalidation mechanism can kick in immediately.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 04:11 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
2024-04-04 04:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-04 08:01 ` Masahiko Sawada <[email protected]>
2024-04-04 08:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Masahiko Sawada @ 2024-04-04 08:01 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Amit Kapila <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Apr 4, 2024 at 1:34 PM Bharath Rupireddy
<[email protected]> wrote:
>
> On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada <[email protected]> wrote:
> >
> > @@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
> > if (SlotSyncCtx->pid == InvalidPid)
> > {
> > SpinLockRelease(&SlotSyncCtx->mutex);
> > + update_synced_slots_inactive_since();
> > return;
> > }
> > SpinLockRelease(&SlotSyncCtx->mutex);
> > @@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
> > }
> >
> > SpinLockRelease(&SlotSyncCtx->mutex);
> > +
> > + update_synced_slots_inactive_since();
> > }
> >
> > Why do we want to update all synced slots' inactive_since values at
> > shutdown in spite of updating the value every time when releasing the
> > slot? It seems to contradict the fact that inactive_since is updated
> > when releasing or restoring the slot.
>
> It is to get the inactive_since right for the cases where the standby
> is promoted without a restart similar to when a standby is promoted
> with restart in which case the inactive_since is set to current time
> in RestoreSlotFromDisk.
>
> Imagine the slot is synced last time at time t1 and then a few hours
> passed, the standby is promoted without a restart. If we don't set
> inactive_since to current time in this case in ShutDownSlotSync, the
> inactive timeout invalidation mechanism can kick in immediately.
>
Thank you for the explanation! I understood the needs.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 04:11 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
2024-04-04 04:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 08:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
@ 2024-04-04 08:36 ` Amit Kapila <[email protected]>
2024-04-04 09:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Amit Kapila @ 2024-04-04 08:36 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Apr 4, 2024 at 1:32 PM Masahiko Sawada <[email protected]> wrote:
>
> On Thu, Apr 4, 2024 at 1:34 PM Bharath Rupireddy
> <[email protected]> wrote:
> >
> > On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada <[email protected]> wrote:
> > >
> > > @@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
> > > if (SlotSyncCtx->pid == InvalidPid)
> > > {
> > > SpinLockRelease(&SlotSyncCtx->mutex);
> > > + update_synced_slots_inactive_since();
> > > return;
> > > }
> > > SpinLockRelease(&SlotSyncCtx->mutex);
> > > @@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
> > > }
> > >
> > > SpinLockRelease(&SlotSyncCtx->mutex);
> > > +
> > > + update_synced_slots_inactive_since();
> > > }
> > >
> > > Why do we want to update all synced slots' inactive_since values at
> > > shutdown in spite of updating the value every time when releasing the
> > > slot? It seems to contradict the fact that inactive_since is updated
> > > when releasing or restoring the slot.
> >
> > It is to get the inactive_since right for the cases where the standby
> > is promoted without a restart similar to when a standby is promoted
> > with restart in which case the inactive_since is set to current time
> > in RestoreSlotFromDisk.
> >
> > Imagine the slot is synced last time at time t1 and then a few hours
> > passed, the standby is promoted without a restart. If we don't set
> > inactive_since to current time in this case in ShutDownSlotSync, the
> > inactive timeout invalidation mechanism can kick in immediately.
> >
>
> Thank you for the explanation! I understood the needs.
>
Do you want to review the v34_0001* further or shall I proceed with
the commit of the same?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 04:11 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
2024-04-04 04:34 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 08:01 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
2024-04-04 08:36 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
@ 2024-04-04 09:05 ` Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Masahiko Sawada @ 2024-04-04 09:05 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Bharath Rupireddy <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Apr 4, 2024 at 5:36 PM Amit Kapila <[email protected]> wrote:
>
> On Thu, Apr 4, 2024 at 1:32 PM Masahiko Sawada <[email protected]> wrote:
> >
> > On Thu, Apr 4, 2024 at 1:34 PM Bharath Rupireddy
> > <[email protected]> wrote:
> > >
> > > On Thu, Apr 4, 2024 at 9:42 AM Masahiko Sawada <[email protected]> wrote:
> > > >
> > > > @@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
> > > > if (SlotSyncCtx->pid == InvalidPid)
> > > > {
> > > > SpinLockRelease(&SlotSyncCtx->mutex);
> > > > + update_synced_slots_inactive_since();
> > > > return;
> > > > }
> > > > SpinLockRelease(&SlotSyncCtx->mutex);
> > > > @@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
> > > > }
> > > >
> > > > SpinLockRelease(&SlotSyncCtx->mutex);
> > > > +
> > > > + update_synced_slots_inactive_since();
> > > > }
> > > >
> > > > Why do we want to update all synced slots' inactive_since values at
> > > > shutdown in spite of updating the value every time when releasing the
> > > > slot? It seems to contradict the fact that inactive_since is updated
> > > > when releasing or restoring the slot.
> > >
> > > It is to get the inactive_since right for the cases where the standby
> > > is promoted without a restart similar to when a standby is promoted
> > > with restart in which case the inactive_since is set to current time
> > > in RestoreSlotFromDisk.
> > >
> > > Imagine the slot is synced last time at time t1 and then a few hours
> > > passed, the standby is promoted without a restart. If we don't set
> > > inactive_since to current time in this case in ShutDownSlotSync, the
> > > inactive timeout invalidation mechanism can kick in immediately.
> > >
> >
> > Thank you for the explanation! I understood the needs.
> >
>
> Do you want to review the v34_0001* further or shall I proceed with
> the commit of the same?
Thanks for asking. The v34-0001 patch looks good to me.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-04 05:18 ` Amit Kapila <[email protected]>
2024-04-04 05:42 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2 siblings, 1 reply; 98+ messages in thread
From: Amit Kapila @ 2024-04-04 05:18 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Apr 3, 2024 at 8:28 PM Bharath Rupireddy
<[email protected]> wrote:
>
> On Wed, Apr 3, 2024 at 6:46 PM Bertrand Drouvot
> <[email protected]> wrote:
> >
> > Just one comment on v32-0001:
> >
> > +# Synced slot on the standby must get its own inactive_since.
> > +is( $standby1->safe_psql(
> > + 'postgres',
> > + "SELECT '$inactive_since_on_primary'::timestamptz <= '$inactive_since_on_standby'::timestamptz AND
> > + '$inactive_since_on_standby'::timestamptz <= '$slot_sync_time'::timestamptz;"
> > + ),
> > + "t",
> > + 'synchronized slot has got its own inactive_since');
> > +
> >
> > By using <= we are not testing that it must get its own inactive_since (as we
> > allow them to be equal in the test). I think we should just add some usleep()
> > where appropriate and deny equality during the tests on inactive_since.
>
> Thanks. It looks like we can ignore the equality in all of the
> inactive_since comparisons. IIUC, all the TAP tests do run with
> primary and standbys on the single BF animals. And, it looks like
> assigning the inactive_since timestamps to perl variables is giving
> the microseconds precision level
> (./tmp_check/log/regress_log_040_standby_failover_slots_sync:inactive_since
> 2024-04-03 14:30:09.691648+00). FWIW, we already have some TAP and SQL
> tests relying on stats_reset timestamps without equality. So, I've
> left the equality for the inactive_since tests.
>
> > Except for the above, v32-0001 LGTM.
>
> Thanks. Please see the attached v33-0001 patch after removing equality
> on inactive_since TAP tests.
>
The v33-0001 looks good to me. I have made minor changes in the
comments/commit message and removed one part of the test which was a
bit confusing and didn't seem to add much value. Let me know what you
think of the attached?
--
With Regards,
Amit Kapila.
Attachments:
[application/octet-stream] v34-0001-Allow-synced-slots-to-have-their-inactive_since.patch (13.8K, ../../CAA4eK1+0uiKrLMNT9C47fEW8=M8-BLFygsFRCOWcjEiEbayXeQ@mail.gmail.com/2-v34-0001-Allow-synced-slots-to-have-their-inactive_since.patch)
download | inline diff:
From fd3aef5c67fb1ba361b9d1a70a47ffadf125b8d0 Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Wed, 3 Apr 2024 14:38:38 +0000
Subject: [PATCH v34] Allow synced slots to have their inactive_since.
This commit does two things:
1) Maintains inactive_since for sync slots whenever the slot is released
just like any other regular slot.
2) Ensures the value is set to the current timestamp during the promotion
of standby to help correctly interpret the time after promotion. Whoever
acquires the slot i.e. makes the slot active will reset it to NULL.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik, Masahiko Sawada
Discussion: https://postgr.es/m/CAA4eK1KrPGwfZV9LYGidjxHeW+rxJ=E2ThjXvwRGLO=iLNuo=Q@mail.gmail.com
Discussion: https://postgr.es/m/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://postgr.es/m/CA+Tgmob_Ta-t2ty8QrKHBGnNLrf4ZYcwhGHGFsuUoFrAEDw4sA@mail.gmail.com
---
doc/src/sgml/system-views.sgml | 7 +++
src/backend/replication/logical/slotsync.c | 48 ++++++++++++++++
src/backend/replication/slot.c | 22 +++-----
src/test/perl/PostgreSQL/Test/Cluster.pm | 31 ++++++++++
src/test/recovery/t/019_replslot_limit.pl | 26 +--------
.../t/040_standby_failover_slots_sync.pl | 56 +++++++++++++++++++
6 files changed, 151 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 3c8dca8ca3..7ed617170f 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2530,6 +2530,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<para>
The time since the slot has become inactive.
<literal>NULL</literal> if the slot is currently being used.
+ Note that for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>), the
+ <structfield>inactive_since</structfield> indicates the last
+ synchronization (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>)
+ time.
</para></entry>
</row>
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 9ac847b780..ea770944e9 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -150,6 +150,7 @@ typedef struct RemoteSlot
} RemoteSlot;
static void slotsync_failure_callback(int code, Datum arg);
+static void update_synced_slots_inactive_since(void);
/*
* If necessary, update the local synced slot's metadata based on the data
@@ -584,6 +585,11 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* overwriting 'invalidated' flag to remote_slot's value. See
* InvalidatePossiblyObsoleteSlot() where it invalidates slot directly
* if the slot is not acquired by other processes.
+ *
+ * XXX: If it ever turns out that slot acquire/release is costly for
+ * cases when none of the slot properties is changed then we can do a
+ * pre-check to ensure that at least one of the slot properties is
+ * changed before acquiring the slot.
*/
ReplicationSlotAcquire(remote_slot->name, true);
@@ -1355,6 +1361,45 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
Assert(false);
}
+/*
+ * Update the inactive_since property for synced slots.
+ *
+ * Note that this function is currently called when we shutdown the slot sync
+ * machinery. This helps correctly interpret the inactive_since if the standby
+ * gets promoted without a restart.
+ */
+static void
+update_synced_slots_inactive_since(void)
+{
+ TimestampTz now = 0;
+
+ /* The slot sync worker mustn't be running by now */
+ Assert(SlotSyncCtx->pid == InvalidPid);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ /* Check if it is a synchronized slot */
+ if (s->in_use && s->data.synced)
+ {
+ Assert(SlotIsLogical(s));
+
+ /* Use the same inactive_since time for all the slots. */
+ if (now == 0)
+ now = GetCurrentTimestamp();
+
+ SpinLockAcquire(&s->mutex);
+ s->inactive_since = now;
+ SpinLockRelease(&s->mutex);
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+}
+
/*
* Shut down the slot sync worker.
*/
@@ -1368,6 +1413,7 @@ ShutDownSlotSync(void)
if (SlotSyncCtx->pid == InvalidPid)
{
SpinLockRelease(&SlotSyncCtx->mutex);
+ update_synced_slots_inactive_since();
return;
}
SpinLockRelease(&SlotSyncCtx->mutex);
@@ -1400,6 +1446,8 @@ ShutDownSlotSync(void)
}
SpinLockRelease(&SlotSyncCtx->mutex);
+
+ update_synced_slots_inactive_since();
}
/*
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index d778c0b921..3bddaae022 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -690,13 +690,10 @@ ReplicationSlotRelease(void)
}
/*
- * Set the last inactive time after marking the slot inactive. We don't
- * set it for the slots currently being synced from the primary to the
- * standby because such slots are typically inactive as decoding is not
- * allowed on those.
+ * Set the time since the slot has become inactive. We get the current
+ * time beforehand to avoid system call while holding the spinlock.
*/
- if (!(RecoveryInProgress() && slot->data.synced))
- now = GetCurrentTimestamp();
+ now = GetCurrentTimestamp();
if (slot->data.persistency == RS_PERSISTENT)
{
@@ -2369,16 +2366,11 @@ RestoreSlotFromDisk(const char *name)
slot->active_pid = 0;
/*
- * We set the last inactive time after loading the slot from the disk
- * into memory. Whoever acquires the slot i.e. makes the slot active
- * will reset it. We don't set it for the slots currently being synced
- * from the primary to the standby because such slots are typically
- * inactive as decoding is not allowed on those.
+ * Set the time since the slot has become inactive after loading the
+ * slot from the disk into memory. Whoever acquires the slot i.e.
+ * makes the slot active will reset it.
*/
- if (!(RecoveryInProgress() && slot->data.synced))
- slot->inactive_since = GetCurrentTimestamp();
- else
- slot->inactive_since = 0;
+ slot->inactive_since = GetCurrentTimestamp();
restored = true;
break;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index b08296605c..54e1008ae5 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -3276,6 +3276,37 @@ sub create_logical_slot_on_standby
=pod
+=item $node->validate_slot_inactive_since(self, slot_name, reference_time)
+
+Validate inactive_since value of a given replication slot against the reference
+time and return it.
+
+=cut
+
+sub validate_slot_inactive_since
+{
+ my ($self, $slot_name, $reference_time) = @_;
+ my $name = $self->name;
+
+ my $inactive_since = $self->safe_psql('postgres',
+ qq(SELECT inactive_since FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;)
+ );
+
+ # Check that the inactive_since is sane
+ is($self->safe_psql('postgres',
+ qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND
+ '$inactive_since'::timestamptz > '$reference_time'::timestamptz;]
+ ),
+ 't',
+ "last inactive time for slot $slot_name is valid on node $name")
+ or die "could not validate captured inactive_since for slot $slot_name";
+
+ return $inactive_since;
+}
+
+=pod
+
=item $node->advance_wal(num)
Advance WAL of node by given number of segments.
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index 3b9a306a8b..96b60cedbb 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -443,7 +443,7 @@ $primary4->safe_psql(
# Get inactive_since value after the slot's creation. Note that the slot is
# still inactive till it's used by the standby below.
my $inactive_since =
- capture_and_validate_slot_inactive_since($primary4, $sb4_slot, $slot_creation_time);
+ $primary4->validate_slot_inactive_since($sb4_slot, $slot_creation_time);
$standby4->start;
@@ -502,7 +502,7 @@ $publisher4->safe_psql('postgres',
# Get inactive_since value after the slot's creation. Note that the slot is
# still inactive till it's used by the subscriber below.
$inactive_since =
- capture_and_validate_slot_inactive_since($publisher4, $lsub4_slot, $slot_creation_time);
+ $publisher4->validate_slot_inactive_since($lsub4_slot, $slot_creation_time);
$subscriber4->start;
$subscriber4->safe_psql('postgres',
@@ -540,26 +540,4 @@ is( $publisher4->safe_psql(
$publisher4->stop;
$subscriber4->stop;
-# Capture and validate inactive_since of a given slot.
-sub capture_and_validate_slot_inactive_since
-{
- my ($node, $slot_name, $slot_creation_time) = @_;
-
- my $inactive_since = $node->safe_psql('postgres',
- qq(SELECT inactive_since FROM pg_replication_slots
- WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;)
- );
-
- # Check that the captured time is sane
- is( $node->safe_psql(
- 'postgres',
- qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND
- '$inactive_since'::timestamptz >= '$slot_creation_time'::timestamptz;]
- ),
- 't',
- "last inactive time for an active slot $slot_name is sane");
-
- return $inactive_since;
-}
-
done_testing();
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 869e3d2e91..ea05ea2769 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1');
$subscriber1->init;
$subscriber1->start;
+# Capture the time before the logical failover slot is created on the
+# primary. We later call this publisher as primary anyway.
+my $slot_creation_time_on_primary = $publisher->safe_psql(
+ 'postgres', qq[
+ SELECT current_timestamp;
+]);
+
# Create a slot on the publisher with failover disabled
$publisher->safe_psql('postgres',
"SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);"
@@ -174,6 +181,11 @@ $primary->poll_query_until(
"SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
1);
+# Capture the inactive_since of the slot from the primary. Note that the slot
+# will be inactive since the corresponding subscription is disabled..
+my $inactive_since_on_primary =
+ $primary->validate_slot_inactive_since('lsub1_slot', $slot_creation_time_on_primary);
+
# Wait for the standby to catch up so that the standby is not lagging behind
# the subscriber.
$primary->wait_for_replay_catchup($standby1);
@@ -190,6 +202,18 @@ is( $standby1->safe_psql(
"t",
'logical slots have synced as true on standby');
+# Capture the inactive_since of the synced slot on the standby
+my $inactive_since_on_standby =
+ $standby1->validate_slot_inactive_since('lsub1_slot', $slot_creation_time_on_primary);
+
+# Synced slot on the standby must get its own inactive_since.
+is( $standby1->safe_psql(
+ 'postgres',
+ "SELECT '$inactive_since_on_primary'::timestamptz < '$inactive_since_on_standby'::timestamptz;"
+ ),
+ "t",
+ 'synchronized slot has got its own inactive_since');
+
##################################################
# Test that the synchronized slot will be dropped if the corresponding remote
# slot on the primary server has been dropped.
@@ -237,6 +261,13 @@ is( $standby1->safe_psql(
$standby1->append_conf('postgresql.conf', 'max_slot_wal_keep_size = -1');
$standby1->reload;
+# Capture the time before the logical failover slot is created on the primary.
+# Note that the subscription creates the slot again on the primary.
+$slot_creation_time_on_primary = $publisher->safe_psql(
+ 'postgres', qq[
+ SELECT current_timestamp;
+]);
+
# To ensure that restart_lsn has moved to a recent WAL position, we re-create
# the subscription and the logical slot.
$subscriber1->safe_psql(
@@ -257,6 +288,11 @@ $primary->poll_query_until(
"SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
1);
+# Capture the inactive_since of the slot from the primary. Note that the slot
+# will be inactive since the corresponding subscription is disabled.
+$inactive_since_on_primary =
+ $primary->validate_slot_inactive_since('lsub1_slot', $slot_creation_time_on_primary);
+
# Wait for the standby to catch up so that the standby is not lagging behind
# the subscriber.
$primary->wait_for_replay_catchup($standby1);
@@ -808,8 +844,28 @@ $primary->reload;
$standby1->start;
$primary->wait_for_replay_catchup($standby1);
+# Capture the time before the standby is promoted
+my $promotion_time_on_primary = $standby1->safe_psql(
+ 'postgres', qq[
+ SELECT current_timestamp;
+]);
+
$standby1->promote;
+# Capture the inactive_since of the synced slot after the promotion.
+# The expectation here is that the slot gets its inactive_since as part of the
+# promotion. We do this check before the slot is enabled on the new primary
+# below, otherwise, the slot gets active setting inactive_since to NULL.
+my $inactive_since_on_new_primary =
+ $standby1->validate_slot_inactive_since('lsub1_slot', $promotion_time_on_primary);
+
+is( $standby1->safe_psql(
+ 'postgres',
+ "SELECT '$inactive_since_on_new_primary'::timestamptz > '$inactive_since_on_primary'::timestamptz"
+ ),
+ "t",
+ 'synchronized slot has got its own inactive_since on the new primary after promotion');
+
# Update subscription with the new primary's connection info
my $standby1_conninfo = $standby1->connstr . ' dbname=postgres';
$subscriber1->safe_psql('postgres',
--
2.28.0.windows.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 05:18 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
@ 2024-04-04 05:42 ` Bharath Rupireddy <[email protected]>
2024-04-04 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-04-04 05:42 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Apr 4, 2024 at 10:48 AM Amit Kapila <[email protected]> wrote:
>
> The v33-0001 looks good to me. I have made minor changes in the
> comments/commit message and removed one part of the test which was a
> bit confusing and didn't seem to add much value. Let me know what you
> think of the attached?
Thanks for the changes. v34-0001 LGTM.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 05:18 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-04 05:42 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-04 11:05 ` Amit Kapila <[email protected]>
2024-04-04 12:22 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Amit Kapila @ 2024-04-04 11:05 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Apr 4, 2024 at 11:12 AM Bharath Rupireddy
<[email protected]> wrote:
>
> On Thu, Apr 4, 2024 at 10:48 AM Amit Kapila <[email protected]> wrote:
> >
> > The v33-0001 looks good to me. I have made minor changes in the
> > comments/commit message and removed one part of the test which was a
> > bit confusing and didn't seem to add much value. Let me know what you
> > think of the attached?
>
> Thanks for the changes. v34-0001 LGTM.
>
I was doing a final review before pushing 0001 and found that
'inactive_since' could be set twice during startup after promotion,
once while restoring slots and then via ShutDownSlotSync(). The reason
is that ShutDownSlotSync() will be invoked in normal startup on
primary though it won't do anything apart from setting inactive_since
if we have synced slots. I think you need to check 'StandbyMode' in
update_synced_slots_inactive_since() and return if the same is not
set. We can't use 'InRecovery' flag as that will be set even during
crash recovery.
Can you please test this once unless you don't agree with the above theory?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 05:18 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-04 05:42 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
@ 2024-04-04 12:22 ` Bharath Rupireddy <[email protected]>
2024-04-05 03:09 ` Re: Introduce XID age and inactive timeout based replication slot invalidation shveta malik <[email protected]>
2024-04-22 13:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 98+ messages in thread
From: Bharath Rupireddy @ 2024-04-04 12:22 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Apr 4, 2024 at 4:35 PM Amit Kapila <[email protected]> wrote:
>
> > Thanks for the changes. v34-0001 LGTM.
>
> I was doing a final review before pushing 0001 and found that
> 'inactive_since' could be set twice during startup after promotion,
> once while restoring slots and then via ShutDownSlotSync(). The reason
> is that ShutDownSlotSync() will be invoked in normal startup on
> primary though it won't do anything apart from setting inactive_since
> if we have synced slots. I think you need to check 'StandbyMode' in
> update_synced_slots_inactive_since() and return if the same is not
> set. We can't use 'InRecovery' flag as that will be set even during
> crash recovery.
>
> Can you please test this once unless you don't agree with the above theory?
Nice catch. I've verified that update_synced_slots_inactive_since is
called even for normal server startups/crash recovery. I've added a
check to exit if the StandbyMode isn't set.
Please find the attached v35 patch.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/x-patch] v35-0001-Allow-synced-slots-to-have-their-inactive_since.patch (13.8K, ../../CALj2ACXZoxXQsV44vE5hLxkhr0-v8KdL=VWq_BqbL+HGRpDogQ@mail.gmail.com/2-v35-0001-Allow-synced-slots-to-have-their-inactive_since.patch)
download | inline diff:
From 03a08bd5ab3a305d199d9427491be37d1a1fd61b Mon Sep 17 00:00:00 2001
From: Bharath Rupireddy <[email protected]>
Date: Thu, 4 Apr 2024 12:06:17 +0000
Subject: [PATCH v35] Allow synced slots to have their inactive_since.
This commit does two things:
1) Maintains inactive_since for sync slots whenever the slot is released
just like any other regular slot.
2) Ensures the value is set to the current timestamp during the promotion
of standby to help correctly interpret the time after promotion. Whoever
acquires the slot i.e. makes the slot active will reset it to NULL.
Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Amit Kapila, Shveta Malik, Masahiko Sawada
Discussion: https://postgr.es/m/CAA4eK1KrPGwfZV9LYGidjxHeW+rxJ=E2ThjXvwRGLO=iLNuo=Q@mail.gmail.com
Discussion: https://postgr.es/m/CALj2ACW4aUe-_uFQOjdWCEN-xXoLGhmvRFnL8SNw_TZ5nJe+aw@mail.gmail.com
Discussion: https://postgr.es/m/CA+Tgmob_Ta-t2ty8QrKHBGnNLrf4ZYcwhGHGFsuUoFrAEDw4sA@mail.gmail.com
---
doc/src/sgml/system-views.sgml | 7 +++
src/backend/replication/logical/slotsync.c | 51 +++++++++++++++++
src/backend/replication/slot.c | 22 +++-----
src/test/perl/PostgreSQL/Test/Cluster.pm | 31 ++++++++++
src/test/recovery/t/019_replslot_limit.pl | 26 +--------
.../t/040_standby_failover_slots_sync.pl | 56 +++++++++++++++++++
6 files changed, 154 insertions(+), 39 deletions(-)
diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml
index 3c8dca8ca3..7ed617170f 100644
--- a/doc/src/sgml/system-views.sgml
+++ b/doc/src/sgml/system-views.sgml
@@ -2530,6 +2530,13 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
<para>
The time since the slot has become inactive.
<literal>NULL</literal> if the slot is currently being used.
+ Note that for slots on the standby that are being synced from a
+ primary server (whose <structfield>synced</structfield> field is
+ <literal>true</literal>), the
+ <structfield>inactive_since</structfield> indicates the last
+ synchronization (see
+ <xref linkend="logicaldecoding-replication-slots-synchronization"/>)
+ time.
</para></entry>
</row>
diff --git a/src/backend/replication/logical/slotsync.c b/src/backend/replication/logical/slotsync.c
index 9ac847b780..75e67b966d 100644
--- a/src/backend/replication/logical/slotsync.c
+++ b/src/backend/replication/logical/slotsync.c
@@ -150,6 +150,7 @@ typedef struct RemoteSlot
} RemoteSlot;
static void slotsync_failure_callback(int code, Datum arg);
+static void update_synced_slots_inactive_since(void);
/*
* If necessary, update the local synced slot's metadata based on the data
@@ -584,6 +585,11 @@ synchronize_one_slot(RemoteSlot *remote_slot, Oid remote_dbid)
* overwriting 'invalidated' flag to remote_slot's value. See
* InvalidatePossiblyObsoleteSlot() where it invalidates slot directly
* if the slot is not acquired by other processes.
+ *
+ * XXX: If it ever turns out that slot acquire/release is costly for
+ * cases when none of the slot properties is changed then we can do a
+ * pre-check to ensure that at least one of the slot properties is
+ * changed before acquiring the slot.
*/
ReplicationSlotAcquire(remote_slot->name, true);
@@ -1355,6 +1361,48 @@ ReplSlotSyncWorkerMain(char *startup_data, size_t startup_data_len)
Assert(false);
}
+/*
+ * Update the inactive_since property for synced slots.
+ *
+ * Note that this function is currently called when we shutdown the slot sync
+ * machinery during standby promotion. This helps correctly interpret the
+ * inactive_since if the standby gets promoted without a restart.
+ */
+static void
+update_synced_slots_inactive_since(void)
+{
+ TimestampTz now = 0;
+
+ if (!StandbyMode)
+ return;
+
+ /* The slot sync worker mustn't be running by now */
+ Assert(SlotSyncCtx->pid == InvalidPid);
+
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+
+ for (int i = 0; i < max_replication_slots; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ /* Check if it is a synchronized slot */
+ if (s->in_use && s->data.synced)
+ {
+ Assert(SlotIsLogical(s));
+
+ /* Use the same inactive_since time for all the slots. */
+ if (now == 0)
+ now = GetCurrentTimestamp();
+
+ SpinLockAcquire(&s->mutex);
+ s->inactive_since = now;
+ SpinLockRelease(&s->mutex);
+ }
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+}
+
/*
* Shut down the slot sync worker.
*/
@@ -1368,6 +1416,7 @@ ShutDownSlotSync(void)
if (SlotSyncCtx->pid == InvalidPid)
{
SpinLockRelease(&SlotSyncCtx->mutex);
+ update_synced_slots_inactive_since();
return;
}
SpinLockRelease(&SlotSyncCtx->mutex);
@@ -1400,6 +1449,8 @@ ShutDownSlotSync(void)
}
SpinLockRelease(&SlotSyncCtx->mutex);
+
+ update_synced_slots_inactive_since();
}
/*
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index d778c0b921..3bddaae022 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -690,13 +690,10 @@ ReplicationSlotRelease(void)
}
/*
- * Set the last inactive time after marking the slot inactive. We don't
- * set it for the slots currently being synced from the primary to the
- * standby because such slots are typically inactive as decoding is not
- * allowed on those.
+ * Set the time since the slot has become inactive. We get the current
+ * time beforehand to avoid system call while holding the spinlock.
*/
- if (!(RecoveryInProgress() && slot->data.synced))
- now = GetCurrentTimestamp();
+ now = GetCurrentTimestamp();
if (slot->data.persistency == RS_PERSISTENT)
{
@@ -2369,16 +2366,11 @@ RestoreSlotFromDisk(const char *name)
slot->active_pid = 0;
/*
- * We set the last inactive time after loading the slot from the disk
- * into memory. Whoever acquires the slot i.e. makes the slot active
- * will reset it. We don't set it for the slots currently being synced
- * from the primary to the standby because such slots are typically
- * inactive as decoding is not allowed on those.
+ * Set the time since the slot has become inactive after loading the
+ * slot from the disk into memory. Whoever acquires the slot i.e.
+ * makes the slot active will reset it.
*/
- if (!(RecoveryInProgress() && slot->data.synced))
- slot->inactive_since = GetCurrentTimestamp();
- else
- slot->inactive_since = 0;
+ slot->inactive_since = GetCurrentTimestamp();
restored = true;
break;
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index b08296605c..54e1008ae5 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -3276,6 +3276,37 @@ sub create_logical_slot_on_standby
=pod
+=item $node->validate_slot_inactive_since(self, slot_name, reference_time)
+
+Validate inactive_since value of a given replication slot against the reference
+time and return it.
+
+=cut
+
+sub validate_slot_inactive_since
+{
+ my ($self, $slot_name, $reference_time) = @_;
+ my $name = $self->name;
+
+ my $inactive_since = $self->safe_psql('postgres',
+ qq(SELECT inactive_since FROM pg_replication_slots
+ WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;)
+ );
+
+ # Check that the inactive_since is sane
+ is($self->safe_psql('postgres',
+ qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND
+ '$inactive_since'::timestamptz > '$reference_time'::timestamptz;]
+ ),
+ 't',
+ "last inactive time for slot $slot_name is valid on node $name")
+ or die "could not validate captured inactive_since for slot $slot_name";
+
+ return $inactive_since;
+}
+
+=pod
+
=item $node->advance_wal(num)
Advance WAL of node by given number of segments.
diff --git a/src/test/recovery/t/019_replslot_limit.pl b/src/test/recovery/t/019_replslot_limit.pl
index 3b9a306a8b..96b60cedbb 100644
--- a/src/test/recovery/t/019_replslot_limit.pl
+++ b/src/test/recovery/t/019_replslot_limit.pl
@@ -443,7 +443,7 @@ $primary4->safe_psql(
# Get inactive_since value after the slot's creation. Note that the slot is
# still inactive till it's used by the standby below.
my $inactive_since =
- capture_and_validate_slot_inactive_since($primary4, $sb4_slot, $slot_creation_time);
+ $primary4->validate_slot_inactive_since($sb4_slot, $slot_creation_time);
$standby4->start;
@@ -502,7 +502,7 @@ $publisher4->safe_psql('postgres',
# Get inactive_since value after the slot's creation. Note that the slot is
# still inactive till it's used by the subscriber below.
$inactive_since =
- capture_and_validate_slot_inactive_since($publisher4, $lsub4_slot, $slot_creation_time);
+ $publisher4->validate_slot_inactive_since($lsub4_slot, $slot_creation_time);
$subscriber4->start;
$subscriber4->safe_psql('postgres',
@@ -540,26 +540,4 @@ is( $publisher4->safe_psql(
$publisher4->stop;
$subscriber4->stop;
-# Capture and validate inactive_since of a given slot.
-sub capture_and_validate_slot_inactive_since
-{
- my ($node, $slot_name, $slot_creation_time) = @_;
-
- my $inactive_since = $node->safe_psql('postgres',
- qq(SELECT inactive_since FROM pg_replication_slots
- WHERE slot_name = '$slot_name' AND inactive_since IS NOT NULL;)
- );
-
- # Check that the captured time is sane
- is( $node->safe_psql(
- 'postgres',
- qq[SELECT '$inactive_since'::timestamptz > to_timestamp(0) AND
- '$inactive_since'::timestamptz >= '$slot_creation_time'::timestamptz;]
- ),
- 't',
- "last inactive time for an active slot $slot_name is sane");
-
- return $inactive_since;
-}
-
done_testing();
diff --git a/src/test/recovery/t/040_standby_failover_slots_sync.pl b/src/test/recovery/t/040_standby_failover_slots_sync.pl
index 869e3d2e91..ea05ea2769 100644
--- a/src/test/recovery/t/040_standby_failover_slots_sync.pl
+++ b/src/test/recovery/t/040_standby_failover_slots_sync.pl
@@ -35,6 +35,13 @@ my $subscriber1 = PostgreSQL::Test::Cluster->new('subscriber1');
$subscriber1->init;
$subscriber1->start;
+# Capture the time before the logical failover slot is created on the
+# primary. We later call this publisher as primary anyway.
+my $slot_creation_time_on_primary = $publisher->safe_psql(
+ 'postgres', qq[
+ SELECT current_timestamp;
+]);
+
# Create a slot on the publisher with failover disabled
$publisher->safe_psql('postgres',
"SELECT 'init' FROM pg_create_logical_replication_slot('lsub1_slot', 'pgoutput', false, false, false);"
@@ -174,6 +181,11 @@ $primary->poll_query_until(
"SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
1);
+# Capture the inactive_since of the slot from the primary. Note that the slot
+# will be inactive since the corresponding subscription is disabled..
+my $inactive_since_on_primary =
+ $primary->validate_slot_inactive_since('lsub1_slot', $slot_creation_time_on_primary);
+
# Wait for the standby to catch up so that the standby is not lagging behind
# the subscriber.
$primary->wait_for_replay_catchup($standby1);
@@ -190,6 +202,18 @@ is( $standby1->safe_psql(
"t",
'logical slots have synced as true on standby');
+# Capture the inactive_since of the synced slot on the standby
+my $inactive_since_on_standby =
+ $standby1->validate_slot_inactive_since('lsub1_slot', $slot_creation_time_on_primary);
+
+# Synced slot on the standby must get its own inactive_since.
+is( $standby1->safe_psql(
+ 'postgres',
+ "SELECT '$inactive_since_on_primary'::timestamptz < '$inactive_since_on_standby'::timestamptz;"
+ ),
+ "t",
+ 'synchronized slot has got its own inactive_since');
+
##################################################
# Test that the synchronized slot will be dropped if the corresponding remote
# slot on the primary server has been dropped.
@@ -237,6 +261,13 @@ is( $standby1->safe_psql(
$standby1->append_conf('postgresql.conf', 'max_slot_wal_keep_size = -1');
$standby1->reload;
+# Capture the time before the logical failover slot is created on the primary.
+# Note that the subscription creates the slot again on the primary.
+$slot_creation_time_on_primary = $publisher->safe_psql(
+ 'postgres', qq[
+ SELECT current_timestamp;
+]);
+
# To ensure that restart_lsn has moved to a recent WAL position, we re-create
# the subscription and the logical slot.
$subscriber1->safe_psql(
@@ -257,6 +288,11 @@ $primary->poll_query_until(
"SELECT COUNT(*) FROM pg_catalog.pg_replication_slots WHERE slot_name = 'lsub1_slot' AND active = 'f'",
1);
+# Capture the inactive_since of the slot from the primary. Note that the slot
+# will be inactive since the corresponding subscription is disabled.
+$inactive_since_on_primary =
+ $primary->validate_slot_inactive_since('lsub1_slot', $slot_creation_time_on_primary);
+
# Wait for the standby to catch up so that the standby is not lagging behind
# the subscriber.
$primary->wait_for_replay_catchup($standby1);
@@ -808,8 +844,28 @@ $primary->reload;
$standby1->start;
$primary->wait_for_replay_catchup($standby1);
+# Capture the time before the standby is promoted
+my $promotion_time_on_primary = $standby1->safe_psql(
+ 'postgres', qq[
+ SELECT current_timestamp;
+]);
+
$standby1->promote;
+# Capture the inactive_since of the synced slot after the promotion.
+# The expectation here is that the slot gets its inactive_since as part of the
+# promotion. We do this check before the slot is enabled on the new primary
+# below, otherwise, the slot gets active setting inactive_since to NULL.
+my $inactive_since_on_new_primary =
+ $standby1->validate_slot_inactive_since('lsub1_slot', $promotion_time_on_primary);
+
+is( $standby1->safe_psql(
+ 'postgres',
+ "SELECT '$inactive_since_on_new_primary'::timestamptz > '$inactive_since_on_primary'::timestamptz"
+ ),
+ "t",
+ 'synchronized slot has got its own inactive_since on the new primary after promotion');
+
# Update subscription with the new primary's connection info
my $standby1_conninfo = $standby1->connstr . ' dbname=postgres';
$subscriber1->safe_psql('postgres',
--
2.34.1
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 05:18 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-04 05:42 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-04 12:22 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-05 03:09 ` shveta malik <[email protected]>
1 sibling, 0 replies; 98+ messages in thread
From: shveta malik @ 2024-04-05 03:09 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; Masahiko Sawada <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>; shveta malik <[email protected]>
On Thu, Apr 4, 2024 at 5:53 PM Bharath Rupireddy
<[email protected]> wrote:
>
> On Thu, Apr 4, 2024 at 4:35 PM Amit Kapila <[email protected]> wrote:
> >
> > > Thanks for the changes. v34-0001 LGTM.
> >
> > I was doing a final review before pushing 0001 and found that
> > 'inactive_since' could be set twice during startup after promotion,
> > once while restoring slots and then via ShutDownSlotSync(). The reason
> > is that ShutDownSlotSync() will be invoked in normal startup on
> > primary though it won't do anything apart from setting inactive_since
> > if we have synced slots. I think you need to check 'StandbyMode' in
> > update_synced_slots_inactive_since() and return if the same is not
> > set. We can't use 'InRecovery' flag as that will be set even during
> > crash recovery.
> >
> > Can you please test this once unless you don't agree with the above theory?
>
> Nice catch. I've verified that update_synced_slots_inactive_since is
> called even for normal server startups/crash recovery. I've added a
> check to exit if the StandbyMode isn't set.
>
> Please find the attached v35 patch.
Thanks for the patch. Tested it , works well. Few cosmetic changes needed:
in 040 test file:
1)
# Capture the inactive_since of the slot from the primary. Note that the slot
# will be inactive since the corresponding subscription is disabled..
2 .. at the end. Replace with one.
2)
# Synced slot on the standby must get its own inactive_since.
. not needed in single line comment (to be consistent with
neighbouring comments)
3)
update_synced_slots_inactive_since():
if (!StandbyMode)
return;
It will be good to add comments here.
thanks
Shveta
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 05:18 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-04 05:42 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-04 12:22 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-22 13:50 ` Masahiko Sawada <[email protected]>
2024-04-25 05:41 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
1 sibling, 1 reply; 98+ messages in thread
From: Masahiko Sawada @ 2024-04-22 13:50 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
Hi,
On Thu, Apr 4, 2024 at 9:23 PM Bharath Rupireddy
<[email protected]> wrote:
>
> On Thu, Apr 4, 2024 at 4:35 PM Amit Kapila <[email protected]> wrote:
> >
> > > Thanks for the changes. v34-0001 LGTM.
> >
> > I was doing a final review before pushing 0001 and found that
> > 'inactive_since' could be set twice during startup after promotion,
> > once while restoring slots and then via ShutDownSlotSync(). The reason
> > is that ShutDownSlotSync() will be invoked in normal startup on
> > primary though it won't do anything apart from setting inactive_since
> > if we have synced slots. I think you need to check 'StandbyMode' in
> > update_synced_slots_inactive_since() and return if the same is not
> > set. We can't use 'InRecovery' flag as that will be set even during
> > crash recovery.
> >
> > Can you please test this once unless you don't agree with the above theory?
>
> Nice catch. I've verified that update_synced_slots_inactive_since is
> called even for normal server startups/crash recovery. I've added a
> check to exit if the StandbyMode isn't set.
>
> Please find the attached v35 patch.
>
The documentation says about both 'active' and 'inactive_since'
columns of pg_replication_slots say:
---
active bool
True if this slot is currently actively being used
inactive_since timestamptz
The time since the slot has become inactive. NULL if the slot is
currently being used. Note that for slots on the standby that are
being synced from a primary server (whose synced field is true), the
inactive_since indicates the last synchronization (see Section 47.2.3)
time.
---
When reading the description I thought if 'active' is true,
'inactive_since' is NULL, but it doesn't seem to apply for temporary
slots. Since we don't reset the active_pid field of temporary slots
when the release, the 'active' is still true in the view but
'inactive_since' is not NULL. Do you think we need to mention it in
the documentation?
As for the timeout-based slot invalidation feature, we could end up
invalidating the temporary slots even if they are shown as active,
which could confuse users. Do we want to somehow deal with it?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 05:18 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-04 05:42 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-04 12:22 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-22 13:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
@ 2024-04-25 05:41 ` Bharath Rupireddy <[email protected]>
2024-04-29 04:03 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
0 siblings, 1 reply; 98+ messages in thread
From: Bharath Rupireddy @ 2024-04-25 05:41 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Amit Kapila <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Mon, Apr 22, 2024 at 7:21 PM Masahiko Sawada <[email protected]> wrote:
>
> > Please find the attached v35 patch.
>
> The documentation says about both 'active' and 'inactive_since'
> columns of pg_replication_slots say:
>
> ---
> active bool
> True if this slot is currently actively being used
>
> inactive_since timestamptz
> The time since the slot has become inactive. NULL if the slot is
> currently being used. Note that for slots on the standby that are
> being synced from a primary server (whose synced field is true), the
> inactive_since indicates the last synchronization (see Section 47.2.3)
> time.
> ---
>
> When reading the description I thought if 'active' is true,
> 'inactive_since' is NULL, but it doesn't seem to apply for temporary
> slots.
Right.
> Since we don't reset the active_pid field of temporary slots
> when the release, the 'active' is still true in the view but
> 'inactive_since' is not NULL.
Right. inactive_since is reset whenever the temporary slot is acquired
again within the same backend that created the temporary slot.
> Do you think we need to mention it in
> the documentation?
I think that's the reason we dropped "active" from the statement. It
was earlier "NULL if the slot is currently actively being used.". But,
per Bertrand's comment
https://www.postgresql.org/message-id/ZehE2IJcsetSJMHC%40ip-10-97-1-34.eu-west-3.compute.internal
changed it to ""NULL if the slot is currently being used.".
Temporary slots retain the active = true and active_pid = <pid of the
backend that created it> even when the slot is not being used until
the lifetime of the backend process. We haven't tied active or
active_pid flags to inactive_since, doing so now to represent the
temporary slot behaviour for active and active_pid will confuse users
more. As far as the inactive_since of a slot is concerned, it is set
to 0 when the slot is being used (acquired) and set to current
timestamp when the slot is not being used (released).
> As for the timeout-based slot invalidation feature, we could end up
> invalidating the temporary slots even if they are shown as active,
> which could confuse users. Do we want to somehow deal with it?
Yes. As long as the temporary slot is lying unused holding up
resources for more than the specified
replication_slot_inactive_timeout, it is bound to get invalidated.
This keeps behaviour consistent and less-confusing to the users.
--
Bharath Rupireddy
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 98+ messages in thread
* Re: Introduce XID age and inactive timeout based replication slot invalidation
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 05:18 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-04 05:42 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-04 11:05 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Amit Kapila <[email protected]>
2024-04-04 12:22 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-22 13:50 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Masahiko Sawada <[email protected]>
2024-04-25 05:41 ` Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
@ 2024-04-29 04:03 ` Amit Kapila <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Amit Kapila @ 2024-04-29 04:03 UTC (permalink / raw)
To: Bharath Rupireddy <[email protected]>; +Cc: Masahiko Sawada <[email protected]>; Bertrand Drouvot <[email protected]>; shveta malik <[email protected]>; Nathan Bossart <[email protected]>; PostgreSQL Hackers <[email protected]>
On Thu, Apr 25, 2024 at 11:11 AM Bharath Rupireddy
<[email protected]> wrote:
>
> On Mon, Apr 22, 2024 at 7:21 PM Masahiko Sawada <[email protected]> wrote:
> >
> > > Please find the attached v35 patch.
> >
> > The documentation says about both 'active' and 'inactive_since'
> > columns of pg_replication_slots say:
> >
> > ---
> > active bool
> > True if this slot is currently actively being used
> >
> > inactive_since timestamptz
> > The time since the slot has become inactive. NULL if the slot is
> > currently being used. Note that for slots on the standby that are
> > being synced from a primary server (whose synced field is true), the
> > inactive_since indicates the last synchronization (see Section 47.2.3)
> > time.
> > ---
> >
> > When reading the description I thought if 'active' is true,
> > 'inactive_since' is NULL, but it doesn't seem to apply for temporary
> > slots.
>
> Right.
>
> > Since we don't reset the active_pid field of temporary slots
> > when the release, the 'active' is still true in the view but
> > 'inactive_since' is not NULL.
>
> Right. inactive_since is reset whenever the temporary slot is acquired
> again within the same backend that created the temporary slot.
>
> > Do you think we need to mention it in
> > the documentation?
>
> I think that's the reason we dropped "active" from the statement. It
> was earlier "NULL if the slot is currently actively being used.". But,
> per Bertrand's comment
> https://www.postgresql.org/message-id/ZehE2IJcsetSJMHC%40ip-10-97-1-34.eu-west-3.compute.internal
> changed it to ""NULL if the slot is currently being used.".
>
> Temporary slots retain the active = true and active_pid = <pid of the
> backend that created it> even when the slot is not being used until
> the lifetime of the backend process. We haven't tied active or
> active_pid flags to inactive_since, doing so now to represent the
> temporary slot behaviour for active and active_pid will confuse users
> more.
>
This is true and it's probably easy for us to understand as we
developed this feature but the same may not be true for others. I
wonder if we can be explicit about the difference of
active/inactive_since by adding something like the following for
inactive_since: Note that this field is not related to the active flag
as temporary slots can remain active till the session ends even when
they are not being used.
Sawada-San, do you have any suggestions on the wording?
>
As far as the inactive_since of a slot is concerned, it is set
> to 0 when the slot is being used (acquired) and set to current
> timestamp when the slot is not being used (released).
>
> > As for the timeout-based slot invalidation feature, we could end up
> > invalidating the temporary slots even if they are shown as active,
> > which could confuse users. Do we want to somehow deal with it?
>
> Yes. As long as the temporary slot is lying unused holding up
> resources for more than the specified
> replication_slot_inactive_timeout, it is bound to get invalidated.
> This keeps behaviour consistent and less-confusing to the users.
>
Agreed. We may want to add something in the docs for this to avoid
confusion with the active flag.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 98+ messages in thread
* [PATCH v1 1/1] remove WaitEventCustomCounterData
@ 2026-07-09 02:50 Nathan Bossart <[email protected]>
0 siblings, 0 replies; 98+ messages in thread
From: Nathan Bossart @ 2026-07-09 02:50 UTC (permalink / raw)
---
src/backend/utils/activity/wait_event.c | 29 ++++++-------------------
src/tools/pgindent/typedefs.list | 1 -
2 files changed, 7 insertions(+), 23 deletions(-)
diff --git a/src/backend/utils/activity/wait_event.c b/src/backend/utils/activity/wait_event.c
index 95635c7f56c..7ee6f99fb50 100644
--- a/src/backend/utils/activity/wait_event.c
+++ b/src/backend/utils/activity/wait_event.c
@@ -81,14 +81,7 @@ typedef struct WaitEventCustomEntryByName
/* dynamic allocation counter for custom wait events */
-typedef struct WaitEventCustomCounterData
-{
- int nextId; /* next ID to assign */
- slock_t mutex; /* protects the counter */
-} WaitEventCustomCounterData;
-
-/* pointer to the shared memory */
-static WaitEventCustomCounterData *WaitEventCustomCounter;
+static int *WaitEventCustomCounter;
/* first event ID of custom wait events */
#define WAIT_EVENT_CUSTOM_INITIAL_ID 1
@@ -110,8 +103,8 @@ const ShmemCallbacks WaitEventCustomShmemCallbacks = {
static void
WaitEventCustomShmemRequest(void *arg)
{
- ShmemRequestStruct(.name = "WaitEventCustomCounterData",
- .size = sizeof(WaitEventCustomCounterData),
+ ShmemRequestStruct(.name = "WaitEventCustomCounter",
+ .size = sizeof(int),
.ptr = (void **) &WaitEventCustomCounter,
);
ShmemRequestHash(.name = "WaitEventCustom hash by wait event information",
@@ -134,9 +127,8 @@ WaitEventCustomShmemRequest(void *arg)
static void
WaitEventCustomShmemInit(void *arg)
{
- /* initialize the allocation counter and its spinlock. */
- WaitEventCustomCounter->nextId = WAIT_EVENT_CUSTOM_INITIAL_ID;
- SpinLockInit(&WaitEventCustomCounter->mutex);
+ /* initialize the allocation counter */
+ *WaitEventCustomCounter = WAIT_EVENT_CUSTOM_INITIAL_ID;
}
/*
@@ -221,19 +213,12 @@ WaitEventCustomNew(uint32 classId, const char *wait_event_name)
}
/* Allocate a new event Id */
- SpinLockAcquire(&WaitEventCustomCounter->mutex);
-
- if (WaitEventCustomCounter->nextId >= WAIT_EVENT_CUSTOM_HASH_SIZE)
- {
- SpinLockRelease(&WaitEventCustomCounter->mutex);
+ if (*WaitEventCustomCounter >= WAIT_EVENT_CUSTOM_HASH_SIZE)
ereport(ERROR,
errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
errmsg("too many custom wait events"));
- }
-
- eventId = WaitEventCustomCounter->nextId++;
- SpinLockRelease(&WaitEventCustomCounter->mutex);
+ eventId = (*WaitEventCustomCounter)++;
/* Register the new wait event */
wait_event_info = classId | eventId;
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index ffb413ab612..3174e0e4ab8 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -3418,7 +3418,6 @@ WaitEvent
WaitEventActivity
WaitEventBuffer
WaitEventClient
-WaitEventCustomCounterData
WaitEventCustomEntryByInfo
WaitEventCustomEntryByName
WaitEventIO
--
2.50.1 (Apple Git-155)
--65l7A33W789+zwjF--
^ permalink raw reply [nested|flat] 98+ messages in thread
end of thread, other threads:[~2026-07-09 02:50 UTC | newest]
Thread overview: 98+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-04-03 14:58 Re: Introduce XID age and inactive timeout based replication slot invalidation Bharath Rupireddy <[email protected]>
2024-04-03 16:27 ` Bertrand Drouvot <[email protected]>
2024-04-05 05:51 ` Bharath Rupireddy <[email protected]>
2024-04-05 07:43 ` Bertrand Drouvot <[email protected]>
2024-04-06 06:25 ` Bharath Rupireddy <[email protected]>
2024-04-06 06:48 ` Amit Kapila <[email protected]>
2024-04-06 11:40 ` Bharath Rupireddy <[email protected]>
2024-04-13 04:06 ` Bharath Rupireddy <[email protected]>
2024-06-17 12:25 ` Bharath Rupireddy <[email protected]>
2024-06-17 15:09 ` Nathan Bossart <[email protected]>
2024-06-24 06:00 ` Bharath Rupireddy <[email protected]>
2024-07-09 22:01 ` Nathan Bossart <[email protected]>
2024-08-12 21:32 ` Masahiko Sawada <[email protected]>
2024-08-12 12:17 ` Ajin Cherian <[email protected]>
2024-08-14 03:50 ` Ajin Cherian <[email protected]>
2024-08-26 06:14 ` Bharath Rupireddy <[email protected]>
2024-08-26 11:05 ` Amit Kapila <[email protected]>
2024-08-29 06:01 ` Bharath Rupireddy <[email protected]>
2024-08-30 02:43 ` Peter Smith <[email protected]>
2024-08-31 08:15 ` Bharath Rupireddy <[email protected]>
2024-09-02 08:06 ` Peter Smith <[email protected]>
2024-09-08 11:54 ` Bharath Rupireddy <[email protected]>
2024-09-09 05:23 ` shveta malik <[email protected]>
2024-09-16 09:47 ` Bharath Rupireddy <[email protected]>
2024-09-09 07:40 ` Peter Smith <[email protected]>
2024-09-10 01:34 ` Peter Smith <[email protected]>
2024-09-16 10:01 ` Bharath Rupireddy <[email protected]>
2024-09-16 11:24 ` Amit Kapila <[email protected]>
2024-09-16 17:10 ` Bharath Rupireddy <[email protected]>
2024-09-18 12:10 ` Amit Kapila <[email protected]>
2024-09-17 01:27 ` Peter Smith <[email protected]>
2024-09-18 06:51 ` shveta malik <[email protected]>
2024-09-18 09:19 ` shveta malik <[email protected]>
2024-09-18 10:01 ` shveta malik <[email protected]>
2024-09-19 04:10 ` shveta malik <[email protected]>
2024-11-13 09:30 ` Nisha Moond <[email protected]>
2024-11-13 23:58 ` Peter Smith <[email protected]>
2024-11-19 07:17 ` Nisha Moond <[email protected]>
2024-11-29 12:36 ` Nisha Moond <[email protected]>
2024-11-14 03:44 ` vignesh C <[email protected]>
2024-11-19 07:12 ` Nisha Moond <[email protected]>
2024-11-19 10:23 ` vignesh C <[email protected]>
2024-11-20 07:59 ` vignesh C <[email protected]>
2024-11-21 12:05 ` Nisha Moond <[email protected]>
2024-11-22 12:13 ` vignesh C <[email protected]>
2024-11-28 09:13 ` vignesh C <[email protected]>
2024-11-29 12:36 ` Nisha Moond <[email protected]>
2024-12-03 07:39 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-12-04 09:30 ` Nisha Moond <[email protected]>
2024-12-04 10:26 ` vignesh C <[email protected]>
2024-12-05 01:13 ` Peter Smith <[email protected]>
2024-12-06 05:34 ` vignesh C <[email protected]>
2024-12-10 11:51 ` Nisha Moond <[email protected]>
2024-12-11 02:44 ` Peter Smith <[email protected]>
2024-12-12 04:12 ` vignesh C <[email protected]>
2024-11-27 03:09 ` Peter Smith <[email protected]>
2024-11-27 10:54 ` Nisha Moond <[email protected]>
2024-11-27 21:37 ` Peter Smith <[email protected]>
2024-11-27 23:50 ` Peter Smith <[email protected]>
2024-11-29 12:37 ` Nisha Moond <[email protected]>
2024-11-28 07:59 ` Hayato Kuroda (Fujitsu) <[email protected]>
2024-11-29 12:36 ` Nisha Moond <[email protected]>
2024-11-28 14:10 ` vignesh C <[email protected]>
2024-11-19 07:20 ` Nisha Moond <[email protected]>
2024-11-21 05:06 ` vignesh C <[email protected]>
2024-11-11 12:42 ` Nisha Moond <[email protected]>
2024-11-13 09:35 ` Nisha Moond <[email protected]>
2024-11-07 10:03 ` Nisha Moond <[email protected]>
2024-11-13 09:29 ` vignesh C <[email protected]>
2024-09-02 09:50 ` Amit Kapila <[email protected]>
2024-09-03 06:55 ` Peter Smith <[email protected]>
2024-09-03 09:31 ` shveta malik <[email protected]>
2024-09-04 03:47 ` shveta malik <[email protected]>
2024-09-04 09:18 ` shveta malik <[email protected]>
2024-09-05 03:55 ` Amit Kapila <[email protected]>
2024-09-05 04:00 ` Amit Kapila <[email protected]>
2024-09-09 03:47 ` shveta malik <[email protected]>
2024-09-09 04:56 ` Bharath Rupireddy <[email protected]>
2024-09-09 04:58 ` shveta malik <[email protected]>
2024-09-09 09:34 ` Amit Kapila <[email protected]>
2024-09-09 18:42 ` Bharath Rupireddy <[email protected]>
2024-09-16 03:25 ` Amit Kapila <[email protected]>
2024-09-08 11:55 ` Bharath Rupireddy <[email protected]>
2024-09-02 06:55 ` Amit Kapila <[email protected]>
2024-04-04 04:11 ` Masahiko Sawada <[email protected]>
2024-04-04 04:34 ` Bharath Rupireddy <[email protected]>
2024-04-04 08:01 ` Masahiko Sawada <[email protected]>
2024-04-04 08:36 ` Amit Kapila <[email protected]>
2024-04-04 09:05 ` Masahiko Sawada <[email protected]>
2024-04-04 05:18 ` Amit Kapila <[email protected]>
2024-04-04 05:42 ` Bharath Rupireddy <[email protected]>
2024-04-04 11:05 ` Amit Kapila <[email protected]>
2024-04-04 12:22 ` Bharath Rupireddy <[email protected]>
2024-04-05 03:09 ` shveta malik <[email protected]>
2024-04-22 13:50 ` Masahiko Sawada <[email protected]>
2024-04-25 05:41 ` Bharath Rupireddy <[email protected]>
2024-04-29 04:03 ` Amit Kapila <[email protected]>
2026-07-09 02:50 [PATCH v1 1/1] remove WaitEventCustomCounterData Nathan Bossart <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox