pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: Fix rare instability in recovery TAP test 009_twophase
12+ messages / 1 participants
[nested] [flat]

* pgsql: Fix rare instability in recovery TAP test 009_twophase
@ 2026-03-04 07:32 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-04 07:32 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 009_twophase

The phase of the test where we want to check that 2PC transactions
prepared on a primary can be committed on a promoted standby relied on
an immediate stop of the primary.  This logic has a race condition: it
could be possible that some records (most likely standby snapshot
records) are generated on the primary before it finishes its shutdown,
without the promoted standby know about them.  When the primary is
recycled as new standby, the test could fail because of a timeline fork
as an effect of these extra records.

This fix takes care of the instability by doing a clean stop of the
primary instead of a teardown (aka immediate stop), so as all records
generated on the primary are sent to the promoted standby and flushed
there.  There is no need for a teardown of the primary in this test
scenario: the commit of 2PC transactions on a promoted standby do not
care about the state of the primary, only of the standby.

This race is very hard to hit in practice, even slow buildfarm members
like skink have a very low rate of reproduction.  Alexander Lakhin has
come up with a recipe to improve the reproduction rate a lot:
- Enable -DWAL_DEBUG.
- Patch the bgwriter so as standby snapshots are generated every
milliseconds.
- Run 009_twophase tests under heavy parallelism.

With this method, the failure appears after a couple of iterations.
With the fix in place, I have been able to run more than 50 iterations
of the parallel test sequence, without seeing a failure.

Issue introduced in 30820982b295, due to a copy-pasto coming from the
surrounding tests.  Thanks also to Hayato Kuroda for digging into the
details of the failure.  He has proposed a fix different than the one of
this commit.  Unfortunately, it relied on injection points, feature only
available in v17.  The solution of this commit is simpler, and can be
applied to v14~v16.

Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/b0102688-6d6c-c86a-db79-e0e91d245b1a@gmail.com
Backpatch-through: 14

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/ad8c86d22cbd043cfb48cc3c18e43c4f267b2eb0

Modified Files
--------------
src/test/recovery/t/009_twophase.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 009_twophase
@ 2026-03-04 07:32 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-04 07:32 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 009_twophase

The phase of the test where we want to check that 2PC transactions
prepared on a primary can be committed on a promoted standby relied on
an immediate stop of the primary.  This logic has a race condition: it
could be possible that some records (most likely standby snapshot
records) are generated on the primary before it finishes its shutdown,
without the promoted standby know about them.  When the primary is
recycled as new standby, the test could fail because of a timeline fork
as an effect of these extra records.

This fix takes care of the instability by doing a clean stop of the
primary instead of a teardown (aka immediate stop), so as all records
generated on the primary are sent to the promoted standby and flushed
there.  There is no need for a teardown of the primary in this test
scenario: the commit of 2PC transactions on a promoted standby do not
care about the state of the primary, only of the standby.

This race is very hard to hit in practice, even slow buildfarm members
like skink have a very low rate of reproduction.  Alexander Lakhin has
come up with a recipe to improve the reproduction rate a lot:
- Enable -DWAL_DEBUG.
- Patch the bgwriter so as standby snapshots are generated every
milliseconds.
- Run 009_twophase tests under heavy parallelism.

With this method, the failure appears after a couple of iterations.
With the fix in place, I have been able to run more than 50 iterations
of the parallel test sequence, without seeing a failure.

Issue introduced in 30820982b295, due to a copy-pasto coming from the
surrounding tests.  Thanks also to Hayato Kuroda for digging into the
details of the failure.  He has proposed a fix different than the one of
this commit.  Unfortunately, it relied on injection points, feature only
available in v17.  The solution of this commit is simpler, and can be
applied to v14~v16.

Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/b0102688-6d6c-c86a-db79-e0e91d245b1a@gmail.com
Backpatch-through: 14

Branch
------
REL_18_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/54e0a8fff1450391019f94102ae9517b10a5e454

Modified Files
--------------
src/test/recovery/t/009_twophase.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 009_twophase
@ 2026-03-04 07:32 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-04 07:32 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 009_twophase

The phase of the test where we want to check that 2PC transactions
prepared on a primary can be committed on a promoted standby relied on
an immediate stop of the primary.  This logic has a race condition: it
could be possible that some records (most likely standby snapshot
records) are generated on the primary before it finishes its shutdown,
without the promoted standby know about them.  When the primary is
recycled as new standby, the test could fail because of a timeline fork
as an effect of these extra records.

This fix takes care of the instability by doing a clean stop of the
primary instead of a teardown (aka immediate stop), so as all records
generated on the primary are sent to the promoted standby and flushed
there.  There is no need for a teardown of the primary in this test
scenario: the commit of 2PC transactions on a promoted standby do not
care about the state of the primary, only of the standby.

This race is very hard to hit in practice, even slow buildfarm members
like skink have a very low rate of reproduction.  Alexander Lakhin has
come up with a recipe to improve the reproduction rate a lot:
- Enable -DWAL_DEBUG.
- Patch the bgwriter so as standby snapshots are generated every
milliseconds.
- Run 009_twophase tests under heavy parallelism.

With this method, the failure appears after a couple of iterations.
With the fix in place, I have been able to run more than 50 iterations
of the parallel test sequence, without seeing a failure.

Issue introduced in 30820982b295, due to a copy-pasto coming from the
surrounding tests.  Thanks also to Hayato Kuroda for digging into the
details of the failure.  He has proposed a fix different than the one of
this commit.  Unfortunately, it relied on injection points, feature only
available in v17.  The solution of this commit is simpler, and can be
applied to v14~v16.

Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/b0102688-6d6c-c86a-db79-e0e91d245b1a@gmail.com
Backpatch-through: 14

Branch
------
REL_17_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/c81b10459bb84776ca2a696ae638a6a2a85b4e88

Modified Files
--------------
src/test/recovery/t/009_twophase.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 009_twophase
@ 2026-03-04 07:32 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-04 07:32 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 009_twophase

The phase of the test where we want to check that 2PC transactions
prepared on a primary can be committed on a promoted standby relied on
an immediate stop of the primary.  This logic has a race condition: it
could be possible that some records (most likely standby snapshot
records) are generated on the primary before it finishes its shutdown,
without the promoted standby know about them.  When the primary is
recycled as new standby, the test could fail because of a timeline fork
as an effect of these extra records.

This fix takes care of the instability by doing a clean stop of the
primary instead of a teardown (aka immediate stop), so as all records
generated on the primary are sent to the promoted standby and flushed
there.  There is no need for a teardown of the primary in this test
scenario: the commit of 2PC transactions on a promoted standby do not
care about the state of the primary, only of the standby.

This race is very hard to hit in practice, even slow buildfarm members
like skink have a very low rate of reproduction.  Alexander Lakhin has
come up with a recipe to improve the reproduction rate a lot:
- Enable -DWAL_DEBUG.
- Patch the bgwriter so as standby snapshots are generated every
milliseconds.
- Run 009_twophase tests under heavy parallelism.

With this method, the failure appears after a couple of iterations.
With the fix in place, I have been able to run more than 50 iterations
of the parallel test sequence, without seeing a failure.

Issue introduced in 30820982b295, due to a copy-pasto coming from the
surrounding tests.  Thanks also to Hayato Kuroda for digging into the
details of the failure.  He has proposed a fix different than the one of
this commit.  Unfortunately, it relied on injection points, feature only
available in v17.  The solution of this commit is simpler, and can be
applied to v14~v16.

Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/b0102688-6d6c-c86a-db79-e0e91d245b1a@gmail.com
Backpatch-through: 14

Branch
------
REL_16_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/a7499ff5eb589c9aa530656fc074706e564ac819

Modified Files
--------------
src/test/recovery/t/009_twophase.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 009_twophase
@ 2026-03-04 07:32 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-04 07:32 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 009_twophase

The phase of the test where we want to check that 2PC transactions
prepared on a primary can be committed on a promoted standby relied on
an immediate stop of the primary.  This logic has a race condition: it
could be possible that some records (most likely standby snapshot
records) are generated on the primary before it finishes its shutdown,
without the promoted standby know about them.  When the primary is
recycled as new standby, the test could fail because of a timeline fork
as an effect of these extra records.

This fix takes care of the instability by doing a clean stop of the
primary instead of a teardown (aka immediate stop), so as all records
generated on the primary are sent to the promoted standby and flushed
there.  There is no need for a teardown of the primary in this test
scenario: the commit of 2PC transactions on a promoted standby do not
care about the state of the primary, only of the standby.

This race is very hard to hit in practice, even slow buildfarm members
like skink have a very low rate of reproduction.  Alexander Lakhin has
come up with a recipe to improve the reproduction rate a lot:
- Enable -DWAL_DEBUG.
- Patch the bgwriter so as standby snapshots are generated every
milliseconds.
- Run 009_twophase tests under heavy parallelism.

With this method, the failure appears after a couple of iterations.
With the fix in place, I have been able to run more than 50 iterations
of the parallel test sequence, without seeing a failure.

Issue introduced in 30820982b295, due to a copy-pasto coming from the
surrounding tests.  Thanks also to Hayato Kuroda for digging into the
details of the failure.  He has proposed a fix different than the one of
this commit.  Unfortunately, it relied on injection points, feature only
available in v17.  The solution of this commit is simpler, and can be
applied to v14~v16.

Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/b0102688-6d6c-c86a-db79-e0e91d245b1a@gmail.com
Backpatch-through: 14

Branch
------
REL_15_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/be233b301e60f5f305296d65cf7bcfd4821319ba

Modified Files
--------------
src/test/recovery/t/009_twophase.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 009_twophase
@ 2026-03-04 07:32 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-04 07:32 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 009_twophase

The phase of the test where we want to check that 2PC transactions
prepared on a primary can be committed on a promoted standby relied on
an immediate stop of the primary.  This logic has a race condition: it
could be possible that some records (most likely standby snapshot
records) are generated on the primary before it finishes its shutdown,
without the promoted standby know about them.  When the primary is
recycled as new standby, the test could fail because of a timeline fork
as an effect of these extra records.

This fix takes care of the instability by doing a clean stop of the
primary instead of a teardown (aka immediate stop), so as all records
generated on the primary are sent to the promoted standby and flushed
there.  There is no need for a teardown of the primary in this test
scenario: the commit of 2PC transactions on a promoted standby do not
care about the state of the primary, only of the standby.

This race is very hard to hit in practice, even slow buildfarm members
like skink have a very low rate of reproduction.  Alexander Lakhin has
come up with a recipe to improve the reproduction rate a lot:
- Enable -DWAL_DEBUG.
- Patch the bgwriter so as standby snapshots are generated every
milliseconds.
- Run 009_twophase tests under heavy parallelism.

With this method, the failure appears after a couple of iterations.
With the fix in place, I have been able to run more than 50 iterations
of the parallel test sequence, without seeing a failure.

Issue introduced in 30820982b295, due to a copy-pasto coming from the
surrounding tests.  Thanks also to Hayato Kuroda for digging into the
details of the failure.  He has proposed a fix different than the one of
this commit.  Unfortunately, it relied on injection points, feature only
available in v17.  The solution of this commit is simpler, and can be
applied to v14~v16.

Reported-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/b0102688-6d6c-c86a-db79-e0e91d245b1a@gmail.com
Backpatch-through: 14

Branch
------
REL_14_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/bce98e49e8ace62c1018dc24dec443a50b347cb3

Modified Files
--------------
src/test/recovery/t/009_twophase.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 004_timeline_switch
@ 2026-03-05 01:06 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-05 01:06 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 004_timeline_switch

This fixes a problem similar to ad8c86d22cbd.  In this case, the test
could fail under the following circumstances:
- The primary is stopped with teardown_node(), meaning that it may not
be able to send all its WAL records to standby_1 and standby_2.
- If standby_2 receives more records than standby_1, attempting to
reconnect standby_2 to the promoted standby_1 would fail because of a
timeline fork.

This race condition is fixed with a simple trick: instead of tearing
down the primary, it is stopped cleanly so as all the WAL records of the
primary are received and flushed by both standby_1 and standby_2.  Once
we do that, there is no need for a wait_for_catchup() before stopping
the node.  The test wants to check that a timeline jump can be achieved
when reconnecting a standby to a promoted standby in the same cluster,
hence an immediate stop of the primary is not required.

This failure is harder to reach than the previous instability of
009_twophase, still the buildfarm has been able to detect this failure
at least once.  I have tried Alexander Lakhin's test trick with the
bgwriter and very aggressive standby snapshots, but I could not
reproduce it directly.  It is reachable, as the buildfarm has proved.

Backpatch down to all supported branches, and this problem can lead to
spurious failures in the buildfarm.

Discussion: https://postgr.es/m/493401a8-063f-436a-8287-a235d9e065fc@gmail.com
Backpatch-through: 14

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/ea4744782b3f9d3dd69be20f31f970bee60f5f09

Modified Files
--------------
src/test/recovery/t/004_timeline_switch.pl | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 004_timeline_switch
@ 2026-03-05 01:06 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-05 01:06 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 004_timeline_switch

This fixes a problem similar to ad8c86d22cbd.  In this case, the test
could fail under the following circumstances:
- The primary is stopped with teardown_node(), meaning that it may not
be able to send all its WAL records to standby_1 and standby_2.
- If standby_2 receives more records than standby_1, attempting to
reconnect standby_2 to the promoted standby_1 would fail because of a
timeline fork.

This race condition is fixed with a simple trick: instead of tearing
down the primary, it is stopped cleanly so as all the WAL records of the
primary are received and flushed by both standby_1 and standby_2.  Once
we do that, there is no need for a wait_for_catchup() before stopping
the node.  The test wants to check that a timeline jump can be achieved
when reconnecting a standby to a promoted standby in the same cluster,
hence an immediate stop of the primary is not required.

This failure is harder to reach than the previous instability of
009_twophase, still the buildfarm has been able to detect this failure
at least once.  I have tried Alexander Lakhin's test trick with the
bgwriter and very aggressive standby snapshots, but I could not
reproduce it directly.  It is reachable, as the buildfarm has proved.

Backpatch down to all supported branches, and this problem can lead to
spurious failures in the buildfarm.

Discussion: https://postgr.es/m/493401a8-063f-436a-8287-a235d9e065fc@gmail.com
Backpatch-through: 14

Branch
------
REL_18_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/7185eddf0522b3146ed1ff6e063e8e129e77c706

Modified Files
--------------
src/test/recovery/t/004_timeline_switch.pl | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 004_timeline_switch
@ 2026-03-05 01:06 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-05 01:06 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 004_timeline_switch

This fixes a problem similar to ad8c86d22cbd.  In this case, the test
could fail under the following circumstances:
- The primary is stopped with teardown_node(), meaning that it may not
be able to send all its WAL records to standby_1 and standby_2.
- If standby_2 receives more records than standby_1, attempting to
reconnect standby_2 to the promoted standby_1 would fail because of a
timeline fork.

This race condition is fixed with a simple trick: instead of tearing
down the primary, it is stopped cleanly so as all the WAL records of the
primary are received and flushed by both standby_1 and standby_2.  Once
we do that, there is no need for a wait_for_catchup() before stopping
the node.  The test wants to check that a timeline jump can be achieved
when reconnecting a standby to a promoted standby in the same cluster,
hence an immediate stop of the primary is not required.

This failure is harder to reach than the previous instability of
009_twophase, still the buildfarm has been able to detect this failure
at least once.  I have tried Alexander Lakhin's test trick with the
bgwriter and very aggressive standby snapshots, but I could not
reproduce it directly.  It is reachable, as the buildfarm has proved.

Backpatch down to all supported branches, and this problem can lead to
spurious failures in the buildfarm.

Discussion: https://postgr.es/m/493401a8-063f-436a-8287-a235d9e065fc@gmail.com
Backpatch-through: 14

Branch
------
REL_17_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/6fb8f424be05148b584f0f206612fb35e4b0e107

Modified Files
--------------
src/test/recovery/t/004_timeline_switch.pl | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 004_timeline_switch
@ 2026-03-05 01:06 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-05 01:06 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 004_timeline_switch

This fixes a problem similar to ad8c86d22cbd.  In this case, the test
could fail under the following circumstances:
- The primary is stopped with teardown_node(), meaning that it may not
be able to send all its WAL records to standby_1 and standby_2.
- If standby_2 receives more records than standby_1, attempting to
reconnect standby_2 to the promoted standby_1 would fail because of a
timeline fork.

This race condition is fixed with a simple trick: instead of tearing
down the primary, it is stopped cleanly so as all the WAL records of the
primary are received and flushed by both standby_1 and standby_2.  Once
we do that, there is no need for a wait_for_catchup() before stopping
the node.  The test wants to check that a timeline jump can be achieved
when reconnecting a standby to a promoted standby in the same cluster,
hence an immediate stop of the primary is not required.

This failure is harder to reach than the previous instability of
009_twophase, still the buildfarm has been able to detect this failure
at least once.  I have tried Alexander Lakhin's test trick with the
bgwriter and very aggressive standby snapshots, but I could not
reproduce it directly.  It is reachable, as the buildfarm has proved.

Backpatch down to all supported branches, and this problem can lead to
spurious failures in the buildfarm.

Discussion: https://postgr.es/m/493401a8-063f-436a-8287-a235d9e065fc@gmail.com
Backpatch-through: 14

Branch
------
REL_16_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/8e3c377e284a22f3f0ce773f6141ff9ed3523cf9

Modified Files
--------------
src/test/recovery/t/004_timeline_switch.pl | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 004_timeline_switch
@ 2026-03-05 01:06 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-05 01:06 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 004_timeline_switch

This fixes a problem similar to ad8c86d22cbd.  In this case, the test
could fail under the following circumstances:
- The primary is stopped with teardown_node(), meaning that it may not
be able to send all its WAL records to standby_1 and standby_2.
- If standby_2 receives more records than standby_1, attempting to
reconnect standby_2 to the promoted standby_1 would fail because of a
timeline fork.

This race condition is fixed with a simple trick: instead of tearing
down the primary, it is stopped cleanly so as all the WAL records of the
primary are received and flushed by both standby_1 and standby_2.  Once
we do that, there is no need for a wait_for_catchup() before stopping
the node.  The test wants to check that a timeline jump can be achieved
when reconnecting a standby to a promoted standby in the same cluster,
hence an immediate stop of the primary is not required.

This failure is harder to reach than the previous instability of
009_twophase, still the buildfarm has been able to detect this failure
at least once.  I have tried Alexander Lakhin's test trick with the
bgwriter and very aggressive standby snapshots, but I could not
reproduce it directly.  It is reachable, as the buildfarm has proved.

Backpatch down to all supported branches, and this problem can lead to
spurious failures in the buildfarm.

Discussion: https://postgr.es/m/493401a8-063f-436a-8287-a235d9e065fc@gmail.com
Backpatch-through: 14

Branch
------
REL_15_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/270e7b4ff519de076f3da5b88ebaed8b579ebd3c

Modified Files
--------------
src/test/recovery/t/004_timeline_switch.pl | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread

* pgsql: Fix rare instability in recovery TAP test 004_timeline_switch
@ 2026-03-05 01:06 Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 12+ messages in thread

From: Michael Paquier @ 2026-03-05 01:06 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix rare instability in recovery TAP test 004_timeline_switch

This fixes a problem similar to ad8c86d22cbd.  In this case, the test
could fail under the following circumstances:
- The primary is stopped with teardown_node(), meaning that it may not
be able to send all its WAL records to standby_1 and standby_2.
- If standby_2 receives more records than standby_1, attempting to
reconnect standby_2 to the promoted standby_1 would fail because of a
timeline fork.

This race condition is fixed with a simple trick: instead of tearing
down the primary, it is stopped cleanly so as all the WAL records of the
primary are received and flushed by both standby_1 and standby_2.  Once
we do that, there is no need for a wait_for_catchup() before stopping
the node.  The test wants to check that a timeline jump can be achieved
when reconnecting a standby to a promoted standby in the same cluster,
hence an immediate stop of the primary is not required.

This failure is harder to reach than the previous instability of
009_twophase, still the buildfarm has been able to detect this failure
at least once.  I have tried Alexander Lakhin's test trick with the
bgwriter and very aggressive standby snapshots, but I could not
reproduce it directly.  It is reachable, as the buildfarm has proved.

Backpatch down to all supported branches, and this problem can lead to
spurious failures in the buildfarm.

Discussion: https://postgr.es/m/493401a8-063f-436a-8287-a235d9e065fc@gmail.com
Backpatch-through: 14

Branch
------
REL_14_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/1346794feb776603232e4cd330affe31ab3c4e1e

Modified Files
--------------
src/test/recovery/t/004_timeline_switch.pl | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)



^ permalink  raw  reply  [nested|flat] 12+ messages in thread


end of thread, other threads:[~2026-03-05 01:06 UTC | newest]

Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-03-04 07:32 pgsql: Fix rare instability in recovery TAP test 009_twophase Michael Paquier <michael@paquier.xyz>
2026-03-04 07:32 pgsql: Fix rare instability in recovery TAP test 009_twophase Michael Paquier <michael@paquier.xyz>
2026-03-04 07:32 pgsql: Fix rare instability in recovery TAP test 009_twophase Michael Paquier <michael@paquier.xyz>
2026-03-04 07:32 pgsql: Fix rare instability in recovery TAP test 009_twophase Michael Paquier <michael@paquier.xyz>
2026-03-04 07:32 pgsql: Fix rare instability in recovery TAP test 009_twophase Michael Paquier <michael@paquier.xyz>
2026-03-04 07:32 pgsql: Fix rare instability in recovery TAP test 009_twophase Michael Paquier <michael@paquier.xyz>
2026-03-05 01:06 pgsql: Fix rare instability in recovery TAP test 004_timeline_switch Michael Paquier <michael@paquier.xyz>
2026-03-05 01:06 pgsql: Fix rare instability in recovery TAP test 004_timeline_switch Michael Paquier <michael@paquier.xyz>
2026-03-05 01:06 pgsql: Fix rare instability in recovery TAP test 004_timeline_switch Michael Paquier <michael@paquier.xyz>
2026-03-05 01:06 pgsql: Fix rare instability in recovery TAP test 004_timeline_switch Michael Paquier <michael@paquier.xyz>
2026-03-05 01:06 pgsql: Fix rare instability in recovery TAP test 004_timeline_switch Michael Paquier <michael@paquier.xyz>
2026-03-05 01:06 pgsql: Fix rare instability in recovery TAP test 004_timeline_switch Michael Paquier <michael@paquier.xyz>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox