public inbox for [email protected]
help / color / mirror / Atom feedRE: [Proposal] Add accumulated statistics for wait event
31+ messages / 9 participants
[nested] [flat]
* RE: [Proposal] Add accumulated statistics for wait event
@ 2018-10-04 08:31 Yotsunaga, Naoki <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Yotsunaga, Naoki @ 2018-10-04 08:31 UTC (permalink / raw)
To: 'Michael Paquier' <[email protected]>; Phil Florent <[email protected]>; +Cc: Tomas Vondra <[email protected]>; [email protected] <[email protected]>
On Thu, July 26, 2018 at 1:25 AM, Michael Paquier wrote:
> Even if you have spiky workloads, sampling may miss those, but even with adding counters for each event
> you would need to query the table holding the counters at an insane frequency to be able to perhaps get
> something out of it as you need to do sampling of the counters as well to extract deltas.
Hi, I was wondering why PostgreSQL did not have the number of wait events and wait time that other databases such as Oracle had as a function, and when I was looking for related threads, I got to this thread.
I am a DB beginner, so please tell me. It says that you can find events that are bottlenecks in sampling, but as you saw above, you can not find events shorter than the sampling interval, right?
If this short event has occurred quite a number of times and it was a considerable amount of time in total, can you solve this problem with sampling?
# I have asked, but I could not understand much of the discussion above and I do not know if such a case can exist.
Also, I think that it can be solved by higher the sampling frequency, but the load will be high, right? I think this method is not very practical.
Moreover, I think that it is not implemented due to the reason that sampling is good as described above and because it affects performance.
How about switching the function on / off and implementing with default off?
Do you care about performance degradation during bottleneck investigation?
When investigating the bottleneck, I think that it is better to find the cause even at the expense of performance.
# If you can detect with high frequency sampling, I think that it depends on whether the sampling or the function(the number of wait events and wait time) is high load.
Since I am a DB beginner, I think that it is saying strange things.
I am glad if you tell me.
-----
Naoki Yotsunaga
^ permalink raw reply [nested|flat] 31+ messages in thread
* RE: [Proposal] Add accumulated statistics for wait event
@ 2018-10-04 09:32 Phil Florent <[email protected]>
parent: Yotsunaga, Naoki <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Phil Florent @ 2018-10-04 09:32 UTC (permalink / raw)
To: Yotsunaga, Naoki <[email protected]>; 'Michael Paquier' <[email protected]>; +Cc: Tomas Vondra <[email protected]>; [email protected] <[email protected]>
Hi,
I am a DB beginner, so please tell me. It says that you can find events that are bottlenecks in sampling, but as you saw above, you can not find events shorter than the sampling interval, right?
If an event occurs frequently and if it is reported in pg_stat_activity, you will catch it again and again while sampling, no matter it duration.
Hence you just need to
* Sample the sessions and consider the active ones. You need to know if they are waiting (PostgreSQL now provides detailed wait events) or if they are on the CPU
* Optionally collect information on the system context at the time of sampling (CPU, memory...), it can be provided by many tools like psutil python library for example
If the client application itself provides information it's even more interesting. With something like program/module/action/client_info/sofar/totalwork in application_name you are able to focus directly on different kind of activity. It can give you information like "I/O waits are meaningful for my batch activity but not for my OLTP activity, if my goal is to improve response time for end users I have to consider that."
Best regards
Phil
________________________________
De : Yotsunaga, Naoki <[email protected]>
Envoyé : jeudi 4 octobre 2018 10:31
À : 'Michael Paquier'; Phil Florent
Cc : Tomas Vondra; [email protected]
Objet : RE: [Proposal] Add accumulated statistics for wait event
On Thu, July 26, 2018 at 1:25 AM, Michael Paquier wrote:
> Even if you have spiky workloads, sampling may miss those, but even with adding counters for each event
> you would need to query the table holding the counters at an insane frequency to be able to perhaps get
> something out of it as you need to do sampling of the counters as well to extract deltas.
Hi, I was wondering why PostgreSQL did not have the number of wait events and wait time that other databases such as Oracle had as a function, and when I was looking for related threads, I got to this thread.
I am a DB beginner, so please tell me. It says that you can find events that are bottlenecks in sampling, but as you saw above, you can not find events shorter than the sampling interval, right?
If this short event has occurred quite a number of times and it was a considerable amount of time in total, can you solve this problem with sampling?
# I have asked, but I could not understand much of the discussion above and I do not know if such a case can exist.
Also, I think that it can be solved by higher the sampling frequency, but the load will be high, right? I think this method is not very practical.
Moreover, I think that it is not implemented due to the reason that sampling is good as described above and because it affects performance.
How about switching the function on / off and implementing with default off?
Do you care about performance degradation during bottleneck investigation?
When investigating the bottleneck, I think that it is better to find the cause even at the expense of performance.
# If you can detect with high frequency sampling, I think that it depends on whether the sampling or the function(the number of wait events and wait time) is high load.
Since I am a DB beginner, I think that it is saying strange things.
I am glad if you tell me.
-----
Naoki Yotsunaga
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: [Proposal] Add accumulated statistics for wait event
@ 2018-10-04 10:58 Michael Paquier <[email protected]>
parent: Phil Florent <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Michael Paquier @ 2018-10-04 10:58 UTC (permalink / raw)
To: Phil Florent <[email protected]>; +Cc: Yotsunaga, Naoki <[email protected]>; Tomas Vondra <[email protected]>; [email protected] <[email protected]>
On Thu, Oct 04, 2018 at 09:32:37AM +0000, Phil Florent wrote:
> I am a DB beginner, so please tell me. It says that you can find
> events that are bottlenecks in sampling, but as you saw above, you can
> not find events shorter than the sampling interval, right?
Yes, which is why it would be as simple as making the interval shorter,
still not too short so as it bloats the amount of information fetched
which needs to be stored and afterwards (perhaps) treated for analysis.
This gets rather close to signal processing. A simple image is for
example, assuming that event A happens 100 times in an interval of 1s,
and event B only once in the same interval of 1s, then if the snapshot
interval is only 1s, then in the worst case A would be treated an equal
of B, which would be wrong.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 31+ messages in thread
* RE: [Proposal] Add accumulated statistics for wait event
@ 2018-10-04 12:54 Phil Florent <[email protected]>
parent: Michael Paquier <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Phil Florent @ 2018-10-04 12:54 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Yotsunaga, Naoki <[email protected]>; Tomas Vondra <[email protected]>; [email protected] <[email protected]>
Hi,
It's the same logic with any polling system. An integration calculation using monte-carlo method with only a few points won't be accurate enough and can even be completely wrong etc.
Polling is OK to troubleshoot a problem on the fly but 2 points are not enough. A few seconds are needed to obtain good enough data, e.g 5-10 seconds of polling with a 0.1=>0.01s interval between 2 queries of the activity.
Polling a few seconds while the user is waiting is normally enough to say if a significant part of the waits are on the database. It's very important to know that. With 1 hour of accumulated statistics, a DBA will always see something to fix. But if the user waits 10 seconds on a particular screen and 1 second is spent on the database it often won't directly help.
Polling gives great information with postgreSQL 10 but it was already useful to catch top queries etc. in older versions.
I always check if activity is adequately reported by my tool using known cases. I want to be sure it will report adequately things in real-world troubleshooting sessions. Sometimes there are bugs in my tool, once there was an issue with postgres (pgstat_report_activty() was not called by workers in parallel index creation)
Best regards
Phil
De : Michael Paquier <[email protected]>
Envoyé : jeudi 4 octobre 2018 12:58
À : Phil Florent
Cc : Yotsunaga, Naoki; Tomas Vondra; [email protected]
Objet : Re: [Proposal] Add accumulated statistics for wait event
On Thu, Oct 04, 2018 at 09:32:37AM +0000, Phil Florent wrote:
> I am a DB beginner, so please tell me. It says that you can find
> events that are bottlenecks in sampling, but as you saw above, you can
> not find events shorter than the sampling interval, right?
Yes, which is why it would be as simple as making the interval shorter,
still not too short so as it bloats the amount of information fetched
which needs to be stored and afterwards (perhaps) treated for analysis.
This gets rather close to signal processing. A simple image is for
example, assuming that event A happens 100 times in an interval of 1s,
and event B only once in the same interval of 1s, then if the snapshot
interval is only 1s, then in the worst case A would be treated an equal
of B, which would be wrong.
--
Michael
^ permalink raw reply [nested|flat] 31+ messages in thread
* RE: [Proposal] Add accumulated statistics for wait event
@ 2018-10-05 08:21 Yotsunaga, Naoki <[email protected]>
parent: Phil Florent <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Yotsunaga, Naoki @ 2018-10-05 08:21 UTC (permalink / raw)
To: 'Phil Florent' <[email protected]>; Michael Paquier <[email protected]>; +Cc: Tomas Vondra <[email protected]>; [email protected] <[email protected]>
On Thu, Oct 4, 2018 at 0:54 AM, Phil Florent wrote:
Phil, Michael, I appreciate your polite comments.
I understand as follows.
We can find it if we shorten the sampling interval, but a lot of information comes out.
# The balance is important.
Also, it is not good unless we have enough samples.
And I have to do various other things.
Is my understand correct?
It seems to me that it is difficult for me if my understanding is right.
Is DBA really able to solve bottlenecks with sampling?
# Since I am a beginner, I feel that way. And other people may not feel it difficult.
What I would like to say is that if we have information on the number of wait events and the wait time(like other DB), we can investigate more easily.
Of course, I understand that it also affects performance. So, I suggest a way that it can switch on and off, defaults is off.
-----
Naoki, Yotsunaga.
^ permalink raw reply [nested|flat] 31+ messages in thread
* RE: [Proposal] Add accumulated statistics for wait event
@ 2018-10-29 01:20 Yotsunaga, Naoki <[email protected]>
parent: Yotsunaga, Naoki <[email protected]>
0 siblings, 2 replies; 31+ messages in thread
From: Yotsunaga, Naoki @ 2018-10-29 01:20 UTC (permalink / raw)
To: 'Phil Florent' <[email protected]>; 'Michael Paquier' <[email protected]>; +Cc: 'Tomas Vondra' <[email protected]>; '[email protected]' <[email protected]>
On Thu, Oct 4, 2018 at 8:22 PM, Yotsunaga Naoki wrote:
Hi, I understood and thought of your statistic comment once again. In the case of sampling, is there enough statistic to investigate?
In the case of a long SQL, I think that it is possible to obtain a sufficient sampling number.
However, in the case of about 1 minute of SQL, only 60 samples can be obtained at most.
#Because legard’s comment.
https://www.postgresql.org/message-id/1539158356795-0.post%40n3.nabble.com
Does this sampling number of 60 give information that I really want?
Perhaps it is not to miss the real problem part?
---------------------------------------
Naoki, Yotsunaga.
^ permalink raw reply [nested|flat] 31+ messages in thread
* RE: [Proposal] Add accumulated statistics for wait event
@ 2018-10-29 06:39 legrand legrand <[email protected]>
parent: Yotsunaga, Naoki <[email protected]>
1 sibling, 0 replies; 31+ messages in thread
From: legrand legrand @ 2018-10-29 06:39 UTC (permalink / raw)
To: pgsql-hackers
Hello,
You are right, sampling has to be "tuned" regarding the event(s) you want to
catch.
Sampling of 1 second interval is good with treatments that take hours, and
not enough for a minute or a second analysis.
May I invite you to try it, using PASH-viewer (github) with pgsentinel
(github).
Changing pgsentiel.c sampling from 1 second
rc = WaitLatch(MyLatch, WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
ash_sampling_period * 1000L,PG_WAIT_EXTENSION);
to 1/10 second
rc = WaitLatch(MyLatch, WL_LATCH_SET | WL_TIMEOUT | WL_POSTMASTER_DEATH,
ash_sampling_period * 100L,PG_WAIT_EXTENSION);
seems the good balance for me (for analysis periods from a few seconds to
minutes).
Regards
PAscal
--
Sent from: http://www.postgresql-archive.org/PostgreSQL-hackers-f1928748.html
^ permalink raw reply [nested|flat] 31+ messages in thread
* RE: [Proposal] Add accumulated statistics for wait event
@ 2018-10-29 13:52 Phil Florent <[email protected]>
parent: Yotsunaga, Naoki <[email protected]>
1 sibling, 0 replies; 31+ messages in thread
From: Phil Florent @ 2018-10-29 13:52 UTC (permalink / raw)
To: Yotsunaga, Naoki <[email protected]>; 'Michael Paquier' <[email protected]>; +Cc: 'Tomas Vondra' <[email protected]>; '[email protected]' <[email protected]>
Hi,
Is DBA really able to solve bottlenecks with sampling?
What I would like to say is that if we have information on the number of wait events and the wait time(like other DB), we can investigate more easily.
Yes you will be able to solve bottlenecks with sampling. In interactive mode, a 1s interval is probably too large. I use 0s1 - 0s01 with my tool and it is normally OK. In batch mode I use 1s=>10s. If you want to visualize the results it's easy to use a dedicated tool and bottlenecks will clearly appear .
Since grafana is now able to connect directly to a postgresql source, I use it to display the information collected from pg_stat_activity and psutil ( e.g https://pgphil.ovh/traqueur_dashboard_02.php page is written in french but panels are in english)
Other DB have accumulated statistics but you can notice that sampling is also their most modern method.
E.g Oracle DB : 20 years ago you already had tools like "utlbstat/utlestat" . Then you had "statspack". Those tools were based on accumulated statistics and the reports were based on differences between 2 points. It was useful to solve major problems but it was limited and not precise enough in many cases.
The preferred feature to identify bottlenecks in the Oracle world is now ASH (active session history). It can help with major problems, specific problems AND it can identify short blockages.
Too bad it is licensed as an option of their Enterprise Edition but similar tools exist and they are also based on sampling of the activity.
With the "official" ASH, sampling and archiving are done internally and you have a circular memory zone dedicated to the feature. Hence the overhead is lower but that's all.
The most advanced interactive tool is called "snapper" and it is also based on sampling.
Best regards
Phil
________________________________
De : Yotsunaga, Naoki <[email protected]>
Envoyé : lundi 29 octobre 2018 02:20
À : 'Phil Florent'; 'Michael Paquier'
Cc : 'Tomas Vondra'; '[email protected]'
Objet : RE: [Proposal] Add accumulated statistics for wait event
On Thu, Oct 4, 2018 at 8:22 PM, Yotsunaga Naoki wrote:
Hi, I understood and thought of your statistic comment once again. In the case of sampling, is there enough statistic to investigate?
In the case of a long SQL, I think that it is possible to obtain a sufficient sampling number.
However, in the case of about 1 minute of SQL, only 60 samples can be obtained at most.
#Because legard’s comment.
https://www.postgresql.org/message-id/1539158356795-0.post%40n3.nabble.com
Does this sampling number of 60 give information that I really want?
Perhaps it is not to miss the real problem part?
---------------------------------------
Naoki, Yotsunaga.
^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v3] Adding wait_for_injection_point helper
@ 2024-02-29 08:31 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 31+ messages in thread
From: Bertrand Drouvot @ 2024-02-29 08:31 UTC (permalink / raw)
---
src/test/perl/PostgreSQL/Test/Cluster.pm | 36 +++++++++++++++++++
.../recovery/t/041_checkpoint_at_promote.pl | 8 +----
2 files changed, 37 insertions(+), 7 deletions(-)
65.3% src/test/perl/PostgreSQL/Test/
34.6% src/test/recovery/t/
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 44c1bb5afd..8d92eb3858 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2742,6 +2742,42 @@ sub lsn
=pod
+=item $node->wait_for_injection_point(injection_name, backend_type)
+
+Wait for the backend_type to wait for the injection point name.
+
+=cut
+
+sub wait_for_injection_point
+{
+ my ($self, $injection_name, $backend_type) = @_;
+ my $die_message;
+
+ if (defined($backend_type))
+ {
+ $backend_type = qq('$backend_type');
+ $die_message = "the backend type $backend_type";
+ }
+ else
+ {
+ $backend_type = 'backend_type';
+ $die_message = 'one backend';
+
+ }
+
+ $self->poll_query_until(
+ 'postgres', qq[
+ SELECT count(*) > 0 FROM pg_stat_activity
+ WHERE backend_type = $backend_type AND wait_event = '$injection_name'
+ ])
+ or die
+ qq(timed out waiting for $die_message to wait for the injection point '$injection_name');
+
+ return;
+}
+
+=pod
+
=item $node->wait_for_catchup(standby_name, mode, target_lsn)
Wait for the replication connection with application_name standby_name until
diff --git a/src/test/recovery/t/041_checkpoint_at_promote.pl b/src/test/recovery/t/041_checkpoint_at_promote.pl
index 47381a2c82..3d6faabc0b 100644
--- a/src/test/recovery/t/041_checkpoint_at_promote.pl
+++ b/src/test/recovery/t/041_checkpoint_at_promote.pl
@@ -79,13 +79,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
# Wait until the checkpointer is in the middle of the restart point
# processing, relying on the custom wait event generated in the
# wait callback used in the injection point previously attached.
-ok( $node_standby->poll_query_until(
- 'postgres',
- qq[SELECT count(*) FROM pg_stat_activity
- WHERE backend_type = 'checkpointer' AND wait_event = 'CreateRestartPoint' ;],
- '1'),
- 'checkpointer is waiting in restart point'
-) or die "Timed out while waiting for checkpointer to run restart point";
+$node_standby->wait_for_injection_point('CreateRestartPoint','checkpointer');
# Check the logs that the restart point has started on standby. This is
# optional, but let's be sure.
--
2.34.1
--4OZRU1j/VeNd8/tm--
^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v3] Adding wait_for_injection_point helper
@ 2024-02-29 08:31 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 31+ messages in thread
From: Bertrand Drouvot @ 2024-02-29 08:31 UTC (permalink / raw)
---
src/test/perl/PostgreSQL/Test/Cluster.pm | 36 +++++++++++++++++++
.../recovery/t/041_checkpoint_at_promote.pl | 8 +----
2 files changed, 37 insertions(+), 7 deletions(-)
65.3% src/test/perl/PostgreSQL/Test/
34.6% src/test/recovery/t/
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 44c1bb5afd..8d92eb3858 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2742,6 +2742,42 @@ sub lsn
=pod
+=item $node->wait_for_injection_point(injection_name, backend_type)
+
+Wait for the backend_type to wait for the injection point name.
+
+=cut
+
+sub wait_for_injection_point
+{
+ my ($self, $injection_name, $backend_type) = @_;
+ my $die_message;
+
+ if (defined($backend_type))
+ {
+ $backend_type = qq('$backend_type');
+ $die_message = "the backend type $backend_type";
+ }
+ else
+ {
+ $backend_type = 'backend_type';
+ $die_message = 'one backend';
+
+ }
+
+ $self->poll_query_until(
+ 'postgres', qq[
+ SELECT count(*) > 0 FROM pg_stat_activity
+ WHERE backend_type = $backend_type AND wait_event = '$injection_name'
+ ])
+ or die
+ qq(timed out waiting for $die_message to wait for the injection point '$injection_name');
+
+ return;
+}
+
+=pod
+
=item $node->wait_for_catchup(standby_name, mode, target_lsn)
Wait for the replication connection with application_name standby_name until
diff --git a/src/test/recovery/t/041_checkpoint_at_promote.pl b/src/test/recovery/t/041_checkpoint_at_promote.pl
index 47381a2c82..3d6faabc0b 100644
--- a/src/test/recovery/t/041_checkpoint_at_promote.pl
+++ b/src/test/recovery/t/041_checkpoint_at_promote.pl
@@ -79,13 +79,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
# Wait until the checkpointer is in the middle of the restart point
# processing, relying on the custom wait event generated in the
# wait callback used in the injection point previously attached.
-ok( $node_standby->poll_query_until(
- 'postgres',
- qq[SELECT count(*) FROM pg_stat_activity
- WHERE backend_type = 'checkpointer' AND wait_event = 'CreateRestartPoint' ;],
- '1'),
- 'checkpointer is waiting in restart point'
-) or die "Timed out while waiting for checkpointer to run restart point";
+$node_standby->wait_for_injection_point('CreateRestartPoint','checkpointer');
# Check the logs that the restart point has started on standby. This is
# optional, but let's be sure.
--
2.34.1
--4OZRU1j/VeNd8/tm--
^ permalink raw reply [nested|flat] 31+ messages in thread
* [PATCH v3] Adding wait_for_injection_point helper
@ 2024-02-29 08:31 Bertrand Drouvot <[email protected]>
0 siblings, 0 replies; 31+ messages in thread
From: Bertrand Drouvot @ 2024-02-29 08:31 UTC (permalink / raw)
---
src/test/perl/PostgreSQL/Test/Cluster.pm | 36 +++++++++++++++++++
.../recovery/t/041_checkpoint_at_promote.pl | 8 +----
2 files changed, 37 insertions(+), 7 deletions(-)
65.3% src/test/perl/PostgreSQL/Test/
34.6% src/test/recovery/t/
diff --git a/src/test/perl/PostgreSQL/Test/Cluster.pm b/src/test/perl/PostgreSQL/Test/Cluster.pm
index 44c1bb5afd..8d92eb3858 100644
--- a/src/test/perl/PostgreSQL/Test/Cluster.pm
+++ b/src/test/perl/PostgreSQL/Test/Cluster.pm
@@ -2742,6 +2742,42 @@ sub lsn
=pod
+=item $node->wait_for_injection_point(injection_name, backend_type)
+
+Wait for the backend_type to wait for the injection point name.
+
+=cut
+
+sub wait_for_injection_point
+{
+ my ($self, $injection_name, $backend_type) = @_;
+ my $die_message;
+
+ if (defined($backend_type))
+ {
+ $backend_type = qq('$backend_type');
+ $die_message = "the backend type $backend_type";
+ }
+ else
+ {
+ $backend_type = 'backend_type';
+ $die_message = 'one backend';
+
+ }
+
+ $self->poll_query_until(
+ 'postgres', qq[
+ SELECT count(*) > 0 FROM pg_stat_activity
+ WHERE backend_type = $backend_type AND wait_event = '$injection_name'
+ ])
+ or die
+ qq(timed out waiting for $die_message to wait for the injection point '$injection_name');
+
+ return;
+}
+
+=pod
+
=item $node->wait_for_catchup(standby_name, mode, target_lsn)
Wait for the replication connection with application_name standby_name until
diff --git a/src/test/recovery/t/041_checkpoint_at_promote.pl b/src/test/recovery/t/041_checkpoint_at_promote.pl
index 47381a2c82..3d6faabc0b 100644
--- a/src/test/recovery/t/041_checkpoint_at_promote.pl
+++ b/src/test/recovery/t/041_checkpoint_at_promote.pl
@@ -79,13 +79,7 @@ $node_primary->wait_for_replay_catchup($node_standby);
# Wait until the checkpointer is in the middle of the restart point
# processing, relying on the custom wait event generated in the
# wait callback used in the injection point previously attached.
-ok( $node_standby->poll_query_until(
- 'postgres',
- qq[SELECT count(*) FROM pg_stat_activity
- WHERE backend_type = 'checkpointer' AND wait_event = 'CreateRestartPoint' ;],
- '1'),
- 'checkpointer is waiting in restart point'
-) or die "Timed out while waiting for checkpointer to run restart point";
+$node_standby->wait_for_injection_point('CreateRestartPoint','checkpointer');
# Check the logs that the restart point has started on standby. This is
# optional, but let's be sure.
--
2.34.1
--4OZRU1j/VeNd8/tm--
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-04-30 05:52 Tom Lane <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Tom Lane @ 2025-04-30 05:52 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Dilip Kumar <[email protected]>; vignesh C <[email protected]>; PostgreSQL Hackers <[email protected]>
Amit Kapila <[email protected]> writes:
> Thanks, Dilip and Sawada-San, for the inputs, and Vignesh for the
> patch. I have pushed the change.
Xuneng Zhou pointed out on Discord that the test case added by
7c99dc587 has caused repeated failures in CI --- though oddly,
it's not failed in the buildfarm so far as I can find. The
failures look like
timed out waiting for match: (?^:WARNING: ( [A-Z0-9]+:)? skipped loading publication: tap_pub_3) at /tmp/cirrus-ci-build/src/test/subscription/t/024_add_drop_pub.pl line 103.
I suspect that what is happening is that the "skipped loading
publication" message comes out sooner than the 024 test script
expects, and thus that it could be fixed by moving the
"my $offset = -s $node_publisher->logfile;" line to be just
before the ALTER SUBSCRIPTION command instead of just after it.
Because the "skipped" message is from LoadPublications() which
is fundamentally invoked as a result of cache flushes, it's
hardly astonishing that its timing would be erratic.
However, I can't really prove this because I've been unable
to reproduce the failure locally, except by moving the
"my $offset" assignment further down which is surely cheating.
Thoughts?
regards, tom lane
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-04-30 12:11 Amit Kapila <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Amit Kapila @ 2025-04-30 12:11 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Dilip Kumar <[email protected]>; vignesh C <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, Apr 30, 2025 at 11:22 AM Tom Lane <[email protected]> wrote:
>
> Xuneng Zhou pointed out on Discord that the test case added by
> 7c99dc587 has caused repeated failures in CI --- though oddly,
> it's not failed in the buildfarm so far as I can find. The
> failures look like
>
> timed out waiting for match: (?^:WARNING: ( [A-Z0-9]+:)? skipped loading publication: tap_pub_3) at /tmp/cirrus-ci-build/src/test/subscription/t/024_add_drop_pub.pl line 103.
>
I analyzed the relevant publisher-side CI Logs [1]:
...
2025-04-19 08:24:14.096 UTC [21961][client backend]
[024_add_drop_pub.pl][7/4:0] LOG: statement: INSERT INTO tab_3
values(1)
2025-04-19 08:24:14.098 UTC [21961][client backend]
[024_add_drop_pub.pl][:0] LOG: disconnection: session time:
0:00:00.003 user=postgres database=postgres host=[local]
2025-04-19 08:24:14.108 UTC [21797][walsender] [tap_sub][30/0:0] LOG:
released logical replication slot "tap_sub"
2025-04-19 08:24:14.108 UTC [21797][walsender] [tap_sub][:0] LOG:
disconnection: session time: 0:00:00.329 user=postgres
database=postgres host=[local]
2025-04-19 08:24:14.127 UTC [21979][not initialized] [[unknown]][:0]
LOG: connection received: host=[local]
2025-04-19 08:24:14.128 UTC [21979][walsender] [[unknown]][23/5:0]
LOG: connection authenticated: user="postgres" method=trust
(/tmp/cirrus-ci-build/build/testrun/subscription/024_add_drop_pub/data/t_024_add_drop_pub_publisher_data/pgdata/pg_hba.conf:117)
2025-04-19 08:24:14.128 UTC [21979][walsender] [[unknown]][23/5:0]
LOG: replication connection authorized: user=postgres
application_name=tap_sub
2025-04-19 08:24:14.129 UTC [21979][walsender] [tap_sub][23/6:0] LOG:
statement: SELECT pg_catalog.set_config('search_path', '', false);
2025-04-19 08:24:14.130 UTC [21979][walsender] [tap_sub][23/0:0] LOG:
received replication command: IDENTIFY_SYSTEM
2025-04-19 08:24:14.130 UTC [21979][walsender] [tap_sub][23/0:0]
STATEMENT: IDENTIFY_SYSTEM
2025-04-19 08:24:14.131 UTC [21979][walsender] [tap_sub][23/0:0] LOG:
received replication command: START_REPLICATION SLOT "tap_sub" LOGICAL
0/0 (proto_version '4', streaming 'parallel', origin 'any',
publication_names '"tap_pub_
...
This shows that walsender restarts after the "INSERT INTO tab_3
values(1)" is processed by the previous walsender ("released logical
replication slot "tap_sub"" is after "INSERT INTO tab_3 values(1)").
So, it is possible that the old apply worker has sent the confirmation
of WAL received location after the Insert (due to keep_alive message
handling). So, after the restart, the new walsender will start
processing WAL after the INSERT and wait for the skipped message LOG
timed out.
Considering the above theory is correct, after "ALTER SUBSCRIPTION
tap_sub SET PUBLICATION tap_pub_3", we should wait for the new
walsender to restart. We are already doing the same for a similar case
in 001_rep_changes.pl (See "# check that change of connection string
and/or publication list causes restart of subscription workers. We
check the state along with application_name to ensure that the
walsender is (re)started.).
Unfortunately, I will be away for the rest of the week. In the
meantime, if you or someone else is able to reproduce and fix it, then
good; otherwise, I'll take care of it after I return.
[1] - https://api.cirrus-ci.com/v1/artifact/task/6561639182368768/testrun/build/testrun/subscription/024_a...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-04-30 16:31 vignesh C <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 2 replies; 31+ messages in thread
From: vignesh C @ 2025-04-30 16:31 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: Tom Lane <[email protected]>; Dilip Kumar <[email protected]>; PostgreSQL Hackers <[email protected]>
On Wed, 30 Apr 2025 at 17:41, Amit Kapila <[email protected]> wrote:
>
> On Wed, Apr 30, 2025 at 11:22 AM Tom Lane <[email protected]> wrote:
> >
> > Xuneng Zhou pointed out on Discord that the test case added by
> > 7c99dc587 has caused repeated failures in CI --- though oddly,
> > it's not failed in the buildfarm so far as I can find. The
> > failures look like
> >
> > timed out waiting for match: (?^:WARNING: ( [A-Z0-9]+:)? skipped loading publication: tap_pub_3) at /tmp/cirrus-ci-build/src/test/subscription/t/024_add_drop_pub.pl line 103.
> >
>
> I analyzed the relevant publisher-side CI Logs [1]:
> ...
> 2025-04-19 08:24:14.096 UTC [21961][client backend]
> [024_add_drop_pub.pl][7/4:0] LOG: statement: INSERT INTO tab_3
> values(1)
> 2025-04-19 08:24:14.098 UTC [21961][client backend]
> [024_add_drop_pub.pl][:0] LOG: disconnection: session time:
> 0:00:00.003 user=postgres database=postgres host=[local]
> 2025-04-19 08:24:14.108 UTC [21797][walsender] [tap_sub][30/0:0] LOG:
> released logical replication slot "tap_sub"
> 2025-04-19 08:24:14.108 UTC [21797][walsender] [tap_sub][:0] LOG:
> disconnection: session time: 0:00:00.329 user=postgres
> database=postgres host=[local]
> 2025-04-19 08:24:14.127 UTC [21979][not initialized] [[unknown]][:0]
> LOG: connection received: host=[local]
> 2025-04-19 08:24:14.128 UTC [21979][walsender] [[unknown]][23/5:0]
> LOG: connection authenticated: user="postgres" method=trust
> (/tmp/cirrus-ci-build/build/testrun/subscription/024_add_drop_pub/data/t_024_add_drop_pub_publisher_data/pgdata/pg_hba.conf:117)
> 2025-04-19 08:24:14.128 UTC [21979][walsender] [[unknown]][23/5:0]
> LOG: replication connection authorized: user=postgres
> application_name=tap_sub
> 2025-04-19 08:24:14.129 UTC [21979][walsender] [tap_sub][23/6:0] LOG:
> statement: SELECT pg_catalog.set_config('search_path', '', false);
> 2025-04-19 08:24:14.130 UTC [21979][walsender] [tap_sub][23/0:0] LOG:
> received replication command: IDENTIFY_SYSTEM
> 2025-04-19 08:24:14.130 UTC [21979][walsender] [tap_sub][23/0:0]
> STATEMENT: IDENTIFY_SYSTEM
> 2025-04-19 08:24:14.131 UTC [21979][walsender] [tap_sub][23/0:0] LOG:
> received replication command: START_REPLICATION SLOT "tap_sub" LOGICAL
> 0/0 (proto_version '4', streaming 'parallel', origin 'any',
> publication_names '"tap_pub_
> ...
>
> This shows that walsender restarts after the "INSERT INTO tab_3
> values(1)" is processed by the previous walsender ("released logical
> replication slot "tap_sub"" is after "INSERT INTO tab_3 values(1)").
> So, it is possible that the old apply worker has sent the confirmation
> of WAL received location after the Insert (due to keep_alive message
> handling). So, after the restart, the new walsender will start
> processing WAL after the INSERT and wait for the skipped message LOG
> timed out.
>
> Considering the above theory is correct, after "ALTER SUBSCRIPTION
> tap_sub SET PUBLICATION tap_pub_3", we should wait for the new
> walsender to restart. We are already doing the same for a similar case
> in 001_rep_changes.pl (See "# check that change of connection string
> and/or publication list causes restart of subscription workers. We
> check the state along with application_name to ensure that the
> walsender is (re)started.).
>
> Unfortunately, I will be away for the rest of the week. In the
> meantime, if you or someone else is able to reproduce and fix it, then
> good; otherwise, I'll take care of it after I return.
I agree with your analysis. I was able to reproduce the issue by
delaying the invalidation of the subscription until the walsender
finished decoding the INSERT operation following the ALTER
SUBSCRIPTION through a debugger and using the lsn from the pg_waldump
of the INSERT after the ALTER SUBSCRIPTION. In this scenario, the
confirmed_flush_lsn ends up pointing to a location after the INSERT.
When the invalidation is eventually received and the apply
worker/walsender is restarted, the restarted walsender begins decoding
from that LSN—after the INSERT—which means the "skipped loading
publication" warning is never triggered, causing the test to fail.
Attached is a patch that ensures the walsender process is properly
restarted after ALTER SUBSCRIPTION, preventing this race condition.
Regards,
Vignesh
Attachments:
[text/x-patch] 0001-Fix-race-condition-after-ALTER-SUBSCRIPTION-SET-PUBL.patch (2.0K, ../../CALDaNm3TH3J8fwj+NcdjdN8DZCdrnmm1kzBsHBW2nkN+h6up3A@mail.gmail.com/2-0001-Fix-race-condition-after-ALTER-SUBSCRIPTION-SET-PUBL.patch)
download | inline diff:
From 26c8efbf59a456f4ad8a87b504180449efe1cd69 Mon Sep 17 00:00:00 2001
From: Vignesh <[email protected]>
Date: Wed, 30 Apr 2025 21:38:33 +0530
Subject: [PATCH] Fix race condition after ALTER SUBSCRIPTION SET PUBLICATION
Previously, after executing ALTER SUBSCRIPTION tap_sub SET PUBLICATION, we
did not wait for the new walsender process to restart. As a result, an INSERT
executed immediately after the ALTER could be decoded and the confirmed flush
lsn is advanced. This could cause replication to resume from a point after the
INSERT. In such cases, we miss the expected warning about the missing
publication.
To fix this, we now ensure that the walsender has restarted before continuing
after ALTER SUBSCRIPTION.
---
src/test/subscription/t/024_add_drop_pub.pl | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/src/test/subscription/t/024_add_drop_pub.pl b/src/test/subscription/t/024_add_drop_pub.pl
index b594941c7cb..e995d8b3839 100644
--- a/src/test/subscription/t/024_add_drop_pub.pl
+++ b/src/test/subscription/t/024_add_drop_pub.pl
@@ -91,10 +91,21 @@ is($result, qq(20|1|10), 'check initial data is copied to subscriber');
$node_publisher->safe_psql('postgres', "CREATE TABLE tab_3 (a int)");
$node_subscriber->safe_psql('postgres', "CREATE TABLE tab_3 (a int)");
+my $oldpid = $node_publisher->safe_psql('postgres',
+ "SELECT pid FROM pg_stat_replication WHERE application_name = 'tap_sub' AND state = 'streaming';"
+);
+
# Set the subscription with a missing publication
$node_subscriber->safe_psql('postgres',
"ALTER SUBSCRIPTION tap_sub SET PUBLICATION tap_pub_3");
+# Wait for the walsender to restart after altering the subscription
+$node_publisher->poll_query_until('postgres',
+ "SELECT pid != $oldpid FROM pg_stat_replication WHERE application_name = 'tap_sub' AND state = 'streaming';"
+ )
+ or die
+ "Timed out while waiting for apply worker to restart after altering the subscription";
+
my $offset = -s $node_publisher->logfile;
$node_publisher->safe_psql('postgres',"INSERT INTO tab_3 values(1)");
--
2.43.0
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-02 01:00 Tom Lane <[email protected]>
parent: vignesh C <[email protected]>
1 sibling, 2 replies; 31+ messages in thread
From: Tom Lane @ 2025-05-02 01:00 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; PostgreSQL Hackers <[email protected]>
vignesh C <[email protected]> writes:
> I agree with your analysis. I was able to reproduce the issue by
> delaying the invalidation of the subscription until the walsender
> finished decoding the INSERT operation following the ALTER
> SUBSCRIPTION through a debugger and using the lsn from the pg_waldump
> of the INSERT after the ALTER SUBSCRIPTION.
Can you be a little more specific about how you reproduced this?
I tried inserting sleep() calls in various likely-looking spots
and could not get a failure that way.
regards, tom lane
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-02 03:43 Xuneng Zhou <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 0 replies; 31+ messages in thread
From: Xuneng Zhou @ 2025-05-02 03:43 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: [email protected]; Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; [email protected]
+1, I was unable to reproduce this with lldb, not sure my way is
appropriate or not.
>
> Can you be a little more specific about how you reproduced this?
> I tried inserting sleep() calls in various likely-looking spots
> and could not get a failure that way.
>
> regards, tom lane
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-02 03:53 vignesh C <[email protected]>
parent: Tom Lane <[email protected]>
1 sibling, 1 reply; 31+ messages in thread
From: vignesh C @ 2025-05-02 03:53 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, 2 May 2025 at 06:30, Tom Lane <[email protected]> wrote:
>
> vignesh C <[email protected]> writes:
> > I agree with your analysis. I was able to reproduce the issue by
> > delaying the invalidation of the subscription until the walsender
> > finished decoding the INSERT operation following the ALTER
> > SUBSCRIPTION through a debugger and using the lsn from the pg_waldump
> > of the INSERT after the ALTER SUBSCRIPTION.
>
> Can you be a little more specific about how you reproduced this?
> I tried inserting sleep() calls in various likely-looking spots
> and could not get a failure that way.
Test Steps:
1) Set up logical replication:
Create a publication on the publisher
Create a subscription on the subscriber
2) Create the following table on the publisher:
CREATE TABLE tab_3 (a int);
3) Create the same table on the subscriber:
CREATE TABLE tab_3 (a int);
4) On the subscriber, alter the subscription to refer to a
non-existent publication:
ALTER SUBSCRIPTION sub1 SET PUBLICATION tap_pub_3;
5) Insert data on the publisher:
INSERT INTO tab_3 VALUES (1);
As expected, the publisher logs the following warning in normal case:
2025-05-02 08:56:45.350 IST [516197] WARNING: skipped loading
publication: tap_pub_3
2025-05-02 08:56:45.350 IST [516197] DETAIL: The publication does
not exist at this point in the WAL.
2025-05-02 08:56:45.350 IST [516197] HINT: Create the publication
if it does not exist.
To simulate a delay in subscription invalidation, I modified the
maybe_reread_subscription() function as follows:
diff --git a/src/backend/replication/logical/worker.c
b/src/backend/replication/logical/worker.c
index 4151a4b2a96..0831784aca3 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -3970,6 +3970,10 @@ maybe_reread_subscription(void)
MemoryContext oldctx;
Subscription *newsub;
bool started_tx = false;
+ bool test = true;
+
+ if (test)
+ return;
This change delays the subscription invalidation logic, preventing the
apply worker from detecting the subscription change immediately.
With the patch applied, repeat steps 1–5.
Using pg_waldump, identify the LSN of the insert:
rmgr: Heap len (rec/tot): 59/ 59, tx: 756, lsn:
0/01711848, prev 0/01711810, desc: INSERT+INIT off: 1
rmgr: Transaction len (rec/tot): 46/ 46, tx: 756, lsn:
0/01711888, prev 0/01711848, desc: COMMIT 2025-05-02 09:06:09.400926
IST
Check the confirmed flush LSN from the walsender via gdb by attaching
it to the walsender process
(gdb) p *MyReplicationSlot
...
confirmed_flush = 24241928
(gdb) p /x 24241928
$4 = 0x171e708
Now attach to the apply worker, set a breakpoint at
maybe_reread_subscription, and continue execution. Once control
reaches the function, set test = false. Now it will identify that
subscription is invalidated and restart the apply worker.
As the walsender has already confirmed_flush position after the
insert, causing the newly started apply worker to miss the inserted
row entirely. This leads to the CI failure. This issue can arise when
the walsender advances more quickly than the apply worker is able to
detect and react to the subscription change.
I could not find a simpler way to reproduce this.
Regards,
Vignesh
Attachments:
[text/x-patch] skip_subscription_invalidation.patch (498B, ../../CALDaNm2Q_pfwiCkaV920iXEbh4D=5MmD_tNQm_GRGX6-MsLxoQ@mail.gmail.com/2-skip_subscription_invalidation.patch)
download | inline diff:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 4151a4b2a96..0831784aca3 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -3970,6 +3970,10 @@ maybe_reread_subscription(void)
MemoryContext oldctx;
Subscription *newsub;
bool started_tx = false;
+ bool test = true;
+
+ if (test)
+ return;
/* When cache state is valid there is nothing to do here. */
if (MySubscriptionValid)
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-02 04:41 Xuneng Zhou <[email protected]>
parent: vignesh C <[email protected]>
1 sibling, 2 replies; 31+ messages in thread
From: Xuneng Zhou @ 2025-05-02 04:41 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; [email protected]; [email protected]
Hi,
Is this an expected behavior?
A race between subscriber LSN feedback and publisher subscription change
processing allows the walsender to restart decoding past relevant WAL
records, bypassing the updated subscription rules for those records.
On Wed, 30 Apr 2025 at 17:41, Amit Kapila <[email protected]> wrote:
> >
> > On Wed, Apr 30, 2025 at 11:22 AM Tom Lane <[email protected]> wrote:
> > >
> > > Xuneng Zhou pointed out on Discord that the test case added by
> > > 7c99dc587 has caused repeated failures in CI --- though oddly,
> > > it's not failed in the buildfarm so far as I can find. The
> > > failures look like
> > >
> > > timed out waiting for match: (?^:WARNING: ( [A-Z0-9]+:)? skipped
> loading publication: tap_pub_3) at
> /tmp/cirrus-ci-build/src/test/subscription/t/024_add_drop_pub.pl line 103.
> > >
> >
> > I analyzed the relevant publisher-side CI Logs [1]:
> > ...
> > 2025-04-19 08:24:14.096 UTC [21961][client backend]
> > [024_add_drop_pub.pl][7/4:0] LOG: statement: INSERT INTO tab_3
> > values(1)
> > 2025-04-19 08:24:14.098 UTC [21961][client backend]
> > [024_add_drop_pub.pl][:0] LOG: disconnection: session time:
> > 0:00:00.003 user=postgres database=postgres host=[local]
> > 2025-04-19 08:24:14.108 UTC [21797][walsender] [tap_sub][30/0:0] LOG:
> > released logical replication slot "tap_sub"
> > 2025-04-19 08:24:14.108 UTC [21797][walsender] [tap_sub][:0] LOG:
> > disconnection: session time: 0:00:00.329 user=postgres
> > database=postgres host=[local]
> > 2025-04-19 08:24:14.127 UTC [21979][not initialized] [[unknown]][:0]
> > LOG: connection received: host=[local]
> > 2025-04-19 08:24:14.128 UTC [21979][walsender] [[unknown]][23/5:0]
> > LOG: connection authenticated: user="postgres" method=trust
> >
> (/tmp/cirrus-ci-build/build/testrun/subscription/024_add_drop_pub/data/t_024_add_drop_pub_publisher_data/pgdata/pg_hba.conf:117)
> > 2025-04-19 08:24:14.128 UTC [21979][walsender] [[unknown]][23/5:0]
> > LOG: replication connection authorized: user=postgres
> > application_name=tap_sub
> > 2025-04-19 08:24:14.129 UTC [21979][walsender] [tap_sub][23/6:0] LOG:
> > statement: SELECT pg_catalog.set_config('search_path', '', false);
> > 2025-04-19 08:24:14.130 UTC [21979][walsender] [tap_sub][23/0:0] LOG:
> > received replication command: IDENTIFY_SYSTEM
> > 2025-04-19 08:24:14.130 UTC [21979][walsender] [tap_sub][23/0:0]
> > STATEMENT: IDENTIFY_SYSTEM
> > 2025-04-19 08:24:14.131 UTC [21979][walsender] [tap_sub][23/0:0] LOG:
> > received replication command: START_REPLICATION SLOT "tap_sub" LOGICAL
> > 0/0 (proto_version '4', streaming 'parallel', origin 'any',
> > publication_names '"tap_pub_
> > ...
> >
> > This shows that walsender restarts after the "INSERT INTO tab_3
> > values(1)" is processed by the previous walsender ("released logical
> > replication slot "tap_sub"" is after "INSERT INTO tab_3 values(1)").
> > So, it is possible that the old apply worker has sent the confirmation
> > of WAL received location after the Insert (due to keep_alive message
> > handling). So, after the restart, the new walsender will start
> > processing WAL after the INSERT and wait for the skipped message LOG
> > timed out.
> >
> > Considering the above theory is correct, after "ALTER SUBSCRIPTION
> > tap_sub SET PUBLICATION tap_pub_3", we should wait for the new
> > walsender to restart. We are already doing the same for a similar case
> > in 001_rep_changes.pl (See "# check that change of connection string
> > and/or publication list causes restart of subscription workers. We
> > check the state along with application_name to ensure that the
> > walsender is (re)started.).
> >
> > Unfortunately, I will be away for the rest of the week. In the
> > meantime, if you or someone else is able to reproduce and fix it, then
> > good; otherwise, I'll take care of it after I return.
>
> I agree with your analysis. I was able to reproduce the issue by
> delaying the invalidation of the subscription until the walsender
> finished decoding the INSERT operation following the ALTER
> SUBSCRIPTION through a debugger and using the lsn from the pg_waldump
> of the INSERT after the ALTER SUBSCRIPTION. In this scenario, the
> confirmed_flush_lsn ends up pointing to a location after the INSERT.
> When the invalidation is eventually received and the apply
> worker/walsender is restarted, the restarted walsender begins decoding
> from that LSN—after the INSERT—which means the "skipped loading
> publication" warning is never triggered, causing the test to fail.
>
> Attached is a patch that ensures the walsender process is properly
> restarted after ALTER SUBSCRIPTION, preventing this race condition.
>
> Regards,
> Vignesh
>
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-02 04:46 Tom Lane <[email protected]>
parent: Xuneng Zhou <[email protected]>
1 sibling, 0 replies; 31+ messages in thread
From: Tom Lane @ 2025-05-02 04:46 UTC (permalink / raw)
To: Xuneng Zhou <[email protected]>; +Cc: vignesh C <[email protected]>; Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; [email protected]
Xuneng Zhou <[email protected]> writes:
> Is this an expected behavior?
I'm wondering that too. I don't see how the repro method Vignesh
describes could correspond to a simple timing issue. It smells
like there's a bug here somewhere.
regards, tom lane
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-02 05:22 vignesh C <[email protected]>
parent: Xuneng Zhou <[email protected]>
1 sibling, 2 replies; 31+ messages in thread
From: vignesh C @ 2025-05-02 05:22 UTC (permalink / raw)
To: Xuneng Zhou <[email protected]>; +Cc: Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; [email protected]; [email protected]
On Fri, 2 May 2025 at 10:11, Xuneng Zhou <[email protected]> wrote:
>
> Hi,
> Is this an expected behavior?
>
> A race between subscriber LSN feedback and publisher subscription change processing allows the walsender to restart decoding past relevant WAL records, bypassing the updated subscription rules for those records.
We have three processes involved in this scenario:
A walsender process on the publisher, responsible for decoding and
sending WAL changes.
An apply worker process on the subscriber, which applies the changes.
A session executing the ALTER SUBSCRIPTION command.
Due to the asynchronous nature of these processes, the ALTER
SUBSCRIPTION command may not be immediately observed by the apply
worker. Meanwhile, the walsender may process and decode an INSERT
statement.
If the insert targets a table (e.g., tab_3) that does not belong to
the current publication (pub1), the walsender silently skips
replicating the record and advances its decoding position. This
position is sent in a keepalive message to the subscriber, and since
there are no pending transactions to flush, the apply worker reports
it as the latest received LSN.
Later, when the apply worker eventually detects the subscription
change, it restarts—but by then, the insert has already been skipped
and is no longer eligible for replay, as the table was not part of the
publication (pub1) at the time of decoding.
This race condition arises because the three processes run
independently and may progress at different speeds due to CPU
scheduling or system load.
Thoughts?
Regards,
Vignesh
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-02 10:44 Xuneng Zhou <[email protected]>
parent: vignesh C <[email protected]>
1 sibling, 0 replies; 31+ messages in thread
From: Xuneng Zhou @ 2025-05-02 10:44 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; [email protected]; [email protected]
Yeh, tks for your clarification. I have a basic understanding of it now. I
mean is this considered a bug or design defect in the codebase? If so,
should we prevent it from occuring in general, not just for this specific
test.
vignesh C <[email protected]>
>
> We have three processes involved in this scenario:
> A walsender process on the publisher, responsible for decoding and
> sending WAL changes.
> An apply worker process on the subscriber, which applies the changes.
> A session executing the ALTER SUBSCRIPTION command.
>
> Due to the asynchronous nature of these processes, the ALTER
> SUBSCRIPTION command may not be immediately observed by the apply
> worker. Meanwhile, the walsender may process and decode an INSERT
> statement.
> If the insert targets a table (e.g., tab_3) that does not belong to
> the current publication (pub1), the walsender silently skips
> replicating the record and advances its decoding position. This
> position is sent in a keepalive message to the subscriber, and since
> there are no pending transactions to flush, the apply worker reports
> it as the latest received LSN.
> Later, when the apply worker eventually detects the subscription
> change, it restarts—but by then, the insert has already been skipped
> and is no longer eligible for replay, as the table was not part of the
> publication (pub1) at the time of decoding.
> This race condition arises because the three processes run
> independently and may progress at different speeds due to CPU
> scheduling or system load.
> Thoughts?
>
> Regards,
> Vignesh
>
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-02 16:54 Tom Lane <[email protected]>
parent: vignesh C <[email protected]>
1 sibling, 1 reply; 31+ messages in thread
From: Tom Lane @ 2025-05-02 16:54 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Xuneng Zhou <[email protected]>; Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; [email protected]
vignesh C <[email protected]> writes:
> Due to the asynchronous nature of these processes, the ALTER
> SUBSCRIPTION command may not be immediately observed by the apply
> worker. Meanwhile, the walsender may process and decode an INSERT
> statement.
> If the insert targets a table (e.g., tab_3) that does not belong to
> the current publication (pub1), the walsender silently skips
> replicating the record and advances its decoding position. This
> position is sent in a keepalive message to the subscriber, and since
> there are no pending transactions to flush, the apply worker reports
> it as the latest received LSN.
So this theory presumes that the apply worker receives and reacts to
the keepalive message, yet it has not observed a relevant
subscriber-side catalog update that surely committed before the
keepalive was generated. It's fairly hard to see how that is okay,
because it's at least adjacent to something that must be considered a
bug: applying transmitted data without having observed DDL updates to
the target table. Why is the processing of keepalives laxer than the
processing of data messages?
regards, tom lane
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-04 13:14 vignesh C <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: vignesh C @ 2025-05-04 13:14 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: Amit Kapila <[email protected]>; Dilip Kumar <[email protected]>; PostgreSQL Hackers <[email protected]>
On Fri, 2 May 2025 at 09:23, vignesh C <[email protected]> wrote:
>
> On Fri, 2 May 2025 at 06:30, Tom Lane <[email protected]> wrote:
> >
> > vignesh C <[email protected]> writes:
> > > I agree with your analysis. I was able to reproduce the issue by
> > > delaying the invalidation of the subscription until the walsender
> > > finished decoding the INSERT operation following the ALTER
> > > SUBSCRIPTION through a debugger and using the lsn from the pg_waldump
> > > of the INSERT after the ALTER SUBSCRIPTION.
> >
> > Can you be a little more specific about how you reproduced this?
> > I tried inserting sleep() calls in various likely-looking spots
> > and could not get a failure that way.
>
> Test Steps:
> 1) Set up logical replication:
> Create a publication on the publisher
> Create a subscription on the subscriber
> 2) Create the following table on the publisher:
> CREATE TABLE tab_3 (a int);
> 3) Create the same table on the subscriber:
> CREATE TABLE tab_3 (a int);
> 4) On the subscriber, alter the subscription to refer to a
> non-existent publication:
> ALTER SUBSCRIPTION sub1 SET PUBLICATION tap_pub_3;
> 5) Insert data on the publisher:
> INSERT INTO tab_3 VALUES (1);
>
> As expected, the publisher logs the following warning in normal case:
> 2025-05-02 08:56:45.350 IST [516197] WARNING: skipped loading
> publication: tap_pub_3
> 2025-05-02 08:56:45.350 IST [516197] DETAIL: The publication does
> not exist at this point in the WAL.
> 2025-05-02 08:56:45.350 IST [516197] HINT: Create the publication
> if it does not exist.
>
> To simulate a delay in subscription invalidation, I modified the
> maybe_reread_subscription() function as follows:
> diff --git a/src/backend/replication/logical/worker.c
> b/src/backend/replication/logical/worker.c
> index 4151a4b2a96..0831784aca3 100644
> --- a/src/backend/replication/logical/worker.c
> +++ b/src/backend/replication/logical/worker.c
> @@ -3970,6 +3970,10 @@ maybe_reread_subscription(void)
> MemoryContext oldctx;
> Subscription *newsub;
> bool started_tx = false;
> + bool test = true;
> +
> + if (test)
> + return;
>
> This change delays the subscription invalidation logic, preventing the
> apply worker from detecting the subscription change immediately.
>
> With the patch applied, repeat steps 1–5.
> Using pg_waldump, identify the LSN of the insert:
> rmgr: Heap len (rec/tot): 59/ 59, tx: 756, lsn:
> 0/01711848, prev 0/01711810, desc: INSERT+INIT off: 1
> rmgr: Transaction len (rec/tot): 46/ 46, tx: 756, lsn:
> 0/01711888, prev 0/01711848, desc: COMMIT 2025-05-02 09:06:09.400926
> IST
>
> Check the confirmed flush LSN from the walsender via gdb by attaching
> it to the walsender process
> (gdb) p *MyReplicationSlot
> ...
> confirmed_flush = 24241928
> (gdb) p /x 24241928
> $4 = 0x171e708
>
> Now attach to the apply worker, set a breakpoint at
> maybe_reread_subscription, and continue execution. Once control
> reaches the function, set test = false. Now it will identify that
> subscription is invalidated and restart the apply worker.
>
> As the walsender has already confirmed_flush position after the
> insert, causing the newly started apply worker to miss the inserted
> row entirely. This leads to the CI failure. This issue can arise when
> the walsender advances more quickly than the apply worker is able to
> detect and react to the subscription change.
>
> I could not find a simpler way to reproduce this.
A simpler way to consistently reproduce the issue is to add a 1-second
sleep in the LogicalRepApplyLoop function, just before the call to
WaitLatchOrSocket. This reproduces the test failure consistently for
me. The failure reason is the same as in [1].
[1] - https://www.postgresql.org/message-id/CALDaNm2Q_pfwiCkaV920iXEbh4D%3D5MmD_tNQm_GRGX6-MsLxoQ%40mail.g...
Regards,
Vignesh
Attachments:
[text/x-patch] ci_failure_reproduce.patch (2.2K, ../../CALDaNm27gUnMG5-gdBLnWH_+4G+EZ_78MA2h8fbGPm9o5LjySA@mail.gmail.com/2-ci_failure_reproduce.patch)
download | inline diff:
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 4151a4b2a96..d0056f5655c 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -3702,6 +3702,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
if (last_received < end_lsn)
last_received = end_lsn;
+ elog(LOG, "Send feedback from 1");
send_feedback(last_received, reply_requested, false);
UpdateWorkerStats(last_received, timestamp, true);
}
@@ -3714,6 +3715,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
}
+ elog(LOG, "Send feedback from 2");
/* confirm all writes so far */
send_feedback(last_received, false, false);
@@ -3739,6 +3741,8 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
if (endofstream)
break;
+ sleep(1);
+
/*
* Wait for more data or latch. If we have unflushed transactions,
* wake up after WalWriterDelay to see if they've been flushed yet (in
@@ -3812,6 +3816,7 @@ LogicalRepApplyLoop(XLogRecPtr last_received)
}
}
+ elog(LOG, "Send feedback from 3");
send_feedback(last_received, requestReply, requestReply);
/*
@@ -3910,7 +3915,7 @@ send_feedback(XLogRecPtr recvpos, bool force, bool requestReply)
pq_sendint64(reply_message, now); /* sendTime */
pq_sendbyte(reply_message, requestReply); /* replyRequested */
- elog(DEBUG2, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X",
+ elog(LOG, "sending feedback (force %d) to recv %X/%X, write %X/%X, flush %X/%X",
force,
LSN_FORMAT_ARGS(recvpos),
LSN_FORMAT_ARGS(writepos),
diff --git a/src/backend/replication/walsender.c b/src/backend/replication/walsender.c
index 9fa8beb6103..9896a8d74d5 100644
--- a/src/backend/replication/walsender.c
+++ b/src/backend/replication/walsender.c
@@ -4062,7 +4062,8 @@ pg_stat_get_wal_senders(PG_FUNCTION_ARGS)
static void
WalSndKeepalive(bool requestReply, XLogRecPtr writePtr)
{
- elog(DEBUG2, "sending replication keepalive");
+ elog(LOG, "sending replication keepalive - writePtr %X/%X",
+ LSN_FORMAT_ARGS(writePtr));
/* construct the message... */
resetStringInfo(&output_message);
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-05 06:18 Amit Kapila <[email protected]>
parent: Tom Lane <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Amit Kapila @ 2025-05-05 06:18 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: vignesh C <[email protected]>; Xuneng Zhou <[email protected]>; Dilip Kumar <[email protected]>; [email protected]
On Fri, May 2, 2025 at 10:24 PM Tom Lane <[email protected]> wrote:
>
> vignesh C <[email protected]> writes:
> > Due to the asynchronous nature of these processes, the ALTER
> > SUBSCRIPTION command may not be immediately observed by the apply
> > worker. Meanwhile, the walsender may process and decode an INSERT
> > statement.
> > If the insert targets a table (e.g., tab_3) that does not belong to
> > the current publication (pub1), the walsender silently skips
> > replicating the record and advances its decoding position. This
> > position is sent in a keepalive message to the subscriber, and since
> > there are no pending transactions to flush, the apply worker reports
> > it as the latest received LSN.
>
> So this theory presumes that the apply worker receives and reacts to
> the keepalive message, yet it has not observed a relevant
> subscriber-side catalog update that surely committed before the
> keepalive was generated. It's fairly hard to see how that is okay,
> because it's at least adjacent to something that must be considered a
> bug: applying transmitted data without having observed DDL updates to
> the target table. Why is the processing of keepalives laxer than the
> processing of data messages?
>
Valid question, as of now, we don't have a specific rule about
ordering the processing of keepalives or invalidation messages. The
effect of invalidation messages is realized by calling
maybe_reread_subscription at three different times after accepting
invalidation message, (a) after starting a transaction in
begin_replication_step, (b) in the commit message handling if there is
no data modification happened in that transaction, and (c) when we
don't get any transactions for a while
The (a) ensures we consume any target table change before applying a
new transaction. The other two places ensure that we keep consuming
invalidation messages from time to time.
Now, we can consume invalidation messages during keepalive message
handling and or at some other places, to ensure that we never process
any remote message before consuming an invalidation message. However,
it is not clear to if this is a must kind of thing. We can provide
strict guarantees for ordering of messages from any one of the
servers, but providing it across nodes doesn't sound to be a
must-criterion.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-06 10:03 Xuneng Zhou <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Xuneng Zhou @ 2025-05-06 10:03 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: vignesh C <[email protected]>; [email protected]; Dilip Kumar <[email protected]>; [email protected]
Hi,
A clear benefit of addressing this in code is to ensure that the user sees
the log message, which can be valuable for trouble-shooting—even under race
conditions.
ereport(WARNING,
errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg("skipped loading
publication: %s", pubname),
errdetail("The publication does not
exist at this point in the WAL."),
errhint("Create the publication if
it does not exist."));
The performance impact appears low, assuming the
AcceptInvalidationMessages and maybe_reread_subscription check are
introduced only in the code path that handles keepalive messages requiring
a reply.
>
> > vignesh C <[email protected]> writes:
> > > Due to the asynchronous nature of these processes, the ALTER
> > > SUBSCRIPTION command may not be immediately observed by the apply
> > > worker. Meanwhile, the walsender may process and decode an INSERT
> > > statement.
> > > If the insert targets a table (e.g., tab_3) that does not belong to
> > > the current publication (pub1), the walsender silently skips
> > > replicating the record and advances its decoding position. This
> > > position is sent in a keepalive message to the subscriber, and since
> > > there are no pending transactions to flush, the apply worker reports
> > > it as the latest received LSN.
> >
> > So this theory presumes that the apply worker receives and reacts to
> > the keepalive message, yet it has not observed a relevant
> > subscriber-side catalog update that surely committed before the
> > keepalive was generated. It's fairly hard to see how that is okay,
> > because it's at least adjacent to something that must be considered a
> > bug: applying transmitted data without having observed DDL updates to
> > the target table. Why is the processing of keepalives laxer than the
> > processing of data messages?
> >
>
> Valid question, as of now, we don't have a specific rule about
> ordering the processing of keepalives or invalidation messages. The
> effect of invalidation messages is realized by calling
> maybe_reread_subscription at three different times after accepting
> invalidation message, (a) after starting a transaction in
> begin_replication_step, (b) in the commit message handling if there is
> no data modification happened in that transaction, and (c) when we
> don't get any transactions for a while
>
> The (a) ensures we consume any target table change before applying a
> new transaction. The other two places ensure that we keep consuming
> invalidation messages from time to time.
>
> Now, we can consume invalidation messages during keepalive message
> handling and or at some other places, to ensure that we never process
> any remote message before consuming an invalidation message. However,
> it is not clear to if this is a must kind of thing. We can provide
> strict guarantees for ordering of messages from any one of the
> servers, but providing it across nodes doesn't sound to be a
> must-criterion.
>
> --
> With Regards,
> Amit Kapila.
>
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-06 11:47 Amit Kapila <[email protected]>
parent: Xuneng Zhou <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Amit Kapila @ 2025-05-06 11:47 UTC (permalink / raw)
To: Xuneng Zhou <[email protected]>; +Cc: vignesh C <[email protected]>; [email protected]; Dilip Kumar <[email protected]>; [email protected]
On Tue, May 6, 2025 at 3:33 PM Xuneng Zhou <[email protected]> wrote:
>
> A clear benefit of addressing this in code is to ensure that the user sees the log message, which can be valuable for trouble-shooting—even under race conditions.
>
I don't think we can take that guarantee because if the Insert is
concurrent or slightly before the Alter Subscription command, then
there won't be a guarantee that users will see the skipped LOG
message.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-10 11:45 Amit Kapila <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Amit Kapila @ 2025-05-10 11:45 UTC (permalink / raw)
To: Xuneng Zhou <[email protected]>; +Cc: vignesh C <[email protected]>; [email protected]; Dilip Kumar <[email protected]>; [email protected]
On Tue, May 6, 2025 at 5:17 PM Amit Kapila <[email protected]> wrote:
>
> On Tue, May 6, 2025 at 3:33 PM Xuneng Zhou <[email protected]> wrote:
> >
> > A clear benefit of addressing this in code is to ensure that the user sees the log message, which can be valuable for trouble-shooting—even under race conditions.
> >
>
> I don't think we can take that guarantee because if the Insert is
> concurrent or slightly before the Alter Subscription command, then
> there won't be a guarantee that users will see the skipped LOG
> message.
>
I am planning to proceed with the test-fix proposed by Vignesh [1]
early next week unless we want to discuss more on this issue.
[1] - https://www.postgresql.org/message-id/CALDaNm3TH3J8fwj%2BNcdjdN8DZCdrnmm1kzBsHBW2nkN%2Bh6up3A%40mail...
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-11 15:19 Xuneng Zhou <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Xuneng Zhou @ 2025-05-11 15:19 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Amit Kapila <[email protected]>; [email protected]; Dilip Kumar <[email protected]>; [email protected]
Hi, I was able to reproduce the failure by adding a 1-second sleep in the
LogicalRepApplyLoop function
However, I noticed that the tests under src/test/subscription run
significantly slower— is this normal?
Also, it looks like the patch mentioned in this thread addresses the issue:
https://www.postgresql.org/message-id/CALDaNm2Q_pfwiCkaV920iXEbh4D%3D5MmD_tNQm_GRGX6-MsLxoQ%40mail.g...
>
> A simpler way to consistently reproduce the issue is to add a 1-second
> sleep in the LogicalRepApplyLoop function, just before the call to
> WaitLatchOrSocket. This reproduces the test failure consistently for
> me. The failure reason is the same as in [1].
>
> [1] -
> https://www.postgresql.org/message-id/CALDaNm2Q_pfwiCkaV920iXEbh4D%3D5MmD_tNQm_GRGX6-MsLxoQ%40mail.g...
>
> Regards,
> Vignesh
>
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-12 05:41 Amit Kapila <[email protected]>
parent: Xuneng Zhou <[email protected]>
0 siblings, 1 reply; 31+ messages in thread
From: Amit Kapila @ 2025-05-12 05:41 UTC (permalink / raw)
To: Xuneng Zhou <[email protected]>; +Cc: vignesh C <[email protected]>; [email protected]; Dilip Kumar <[email protected]>; [email protected]
On Sun, May 11, 2025 at 8:49 PM Xuneng Zhou <[email protected]> wrote:
>
> Hi, I was able to reproduce the failure by adding a 1-second sleep in the LogicalRepApplyLoop function
> However, I noticed that the tests under src/test/subscription run significantly slower— is this normal?
>
Yes, because you made apply slower by adding a sleep.
> Also, it looks like the patch mentioned in this thread addresses the issue:
> https://www.postgresql.org/message-id/CALDaNm2Q_pfwiCkaV920iXEbh4D%3D5MmD_tNQm_GRGX6-MsLxoQ%40mail.g...
>>
So you are okay with a test-only fix?
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-12 09:01 Xuneng Zhou <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 31+ messages in thread
From: Xuneng Zhou @ 2025-05-12 09:01 UTC (permalink / raw)
To: Amit Kapila <[email protected]>; +Cc: vignesh C <[email protected]>; [email protected]; Dilip Kumar <[email protected]>; [email protected]
If the presumed theory regarding the cause of the issue is correct — as
outlined in this email
<https://www.postgresql.org/message-id/CALDaNm2Pmbc-7KM3nRgZcq1EBhbdvWJSTie-st57oGuKP4O44w%40mail.gma...;
— and no data replication occurs in this scenario
<https://www.postgresql.org/message-id/CAA4eK1Jz20hnPRDtPDo2BbSt0Xf8u2zY4Tc84R0OAQN8M%3D9iCQ%40mail.g...;
, then the proposed fix seems ok to me.
But I don’t have the expertise to fully assess the trade-offs between
enforcing strict ordering across nodes and maintaining the current behavior.
>
> > Also, it looks like the patch mentioned in this thread addresses the
> issue:
> >
> https://www.postgresql.org/message-id/CALDaNm2Q_pfwiCkaV920iXEbh4D%3D5MmD_tNQm_GRGX6-MsLxoQ%40mail.g...
> >>
>
> So you are okay with a test-only fix?
>
^ permalink raw reply [nested|flat] 31+ messages in thread
* Re: Add an option to skip loading missing publication to avoid logical replication failure
@ 2025-05-13 11:33 Amit Kapila <[email protected]>
parent: Amit Kapila <[email protected]>
0 siblings, 0 replies; 31+ messages in thread
From: Amit Kapila @ 2025-05-13 11:33 UTC (permalink / raw)
To: Xuneng Zhou <[email protected]>; +Cc: vignesh C <[email protected]>; [email protected]; Dilip Kumar <[email protected]>; [email protected]
On Sat, May 10, 2025 at 5:15 PM Amit Kapila <[email protected]> wrote:
>
> I am planning to proceed with the test-fix proposed by Vignesh [1]
> early next week unless we want to discuss more on this issue.
>
Pushed.
> [1] - https://www.postgresql.org/message-id/CALDaNm3TH3J8fwj%2BNcdjdN8DZCdrnmm1kzBsHBW2nkN%2Bh6up3A%40mail...
>
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 31+ messages in thread
end of thread, other threads:[~2025-05-13 11:33 UTC | newest]
Thread overview: 31+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-10-04 08:31 RE: [Proposal] Add accumulated statistics for wait event Yotsunaga, Naoki <[email protected]>
2018-10-04 09:32 ` Phil Florent <[email protected]>
2018-10-04 10:58 ` Michael Paquier <[email protected]>
2018-10-04 12:54 ` Phil Florent <[email protected]>
2018-10-05 08:21 ` Yotsunaga, Naoki <[email protected]>
2018-10-29 01:20 ` Yotsunaga, Naoki <[email protected]>
2018-10-29 06:39 ` legrand legrand <[email protected]>
2018-10-29 13:52 ` Phil Florent <[email protected]>
2024-02-29 08:31 [PATCH v3] Adding wait_for_injection_point helper Bertrand Drouvot <[email protected]>
2024-02-29 08:31 [PATCH v3] Adding wait_for_injection_point helper Bertrand Drouvot <[email protected]>
2024-02-29 08:31 [PATCH v3] Adding wait_for_injection_point helper Bertrand Drouvot <[email protected]>
2025-04-30 05:52 Re: Add an option to skip loading missing publication to avoid logical replication failure Tom Lane <[email protected]>
2025-04-30 12:11 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Amit Kapila <[email protected]>
2025-04-30 16:31 ` Re: Add an option to skip loading missing publication to avoid logical replication failure vignesh C <[email protected]>
2025-05-02 01:00 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Tom Lane <[email protected]>
2025-05-02 03:43 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Xuneng Zhou <[email protected]>
2025-05-02 03:53 ` Re: Add an option to skip loading missing publication to avoid logical replication failure vignesh C <[email protected]>
2025-05-04 13:14 ` Re: Add an option to skip loading missing publication to avoid logical replication failure vignesh C <[email protected]>
2025-05-11 15:19 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Xuneng Zhou <[email protected]>
2025-05-12 05:41 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Amit Kapila <[email protected]>
2025-05-12 09:01 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Xuneng Zhou <[email protected]>
2025-05-02 04:41 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Xuneng Zhou <[email protected]>
2025-05-02 04:46 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Tom Lane <[email protected]>
2025-05-02 05:22 ` Re: Add an option to skip loading missing publication to avoid logical replication failure vignesh C <[email protected]>
2025-05-02 10:44 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Xuneng Zhou <[email protected]>
2025-05-02 16:54 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Tom Lane <[email protected]>
2025-05-05 06:18 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Amit Kapila <[email protected]>
2025-05-06 10:03 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Xuneng Zhou <[email protected]>
2025-05-06 11:47 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Amit Kapila <[email protected]>
2025-05-10 11:45 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Amit Kapila <[email protected]>
2025-05-13 11:33 ` Re: Add an option to skip loading missing publication to avoid logical replication failure Amit Kapila <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox