pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: Fix control file update done in restartpoints still running afte
6+ messages / 1 participants
[nested] [flat]

* pgsql: Fix control file update done in restartpoints still running afte
@ 2022-05-08 23:41  Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2022-05-08 23:41 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix control file update done in restartpoints still running after promotion

If a cluster is promoted (aka the control file shows a state different
than DB_IN_ARCHIVE_RECOVERY) while CreateRestartPoint() is still
processing, this function could miss an update of the control file for
"checkPoint" and "checkPointCopy" but still do the recycling and/or
removal of the past WAL segments, assuming that the to-be-updated LSN
values should be used as reference points for the cleanup.  This causes
a follow-up restart attempting crash recovery to fail with a PANIC on a
missing checkpoint record if the end-of-recovery checkpoint triggered by
the promotion did not complete while the cluster abruptly stopped or
crashed before the completion of this checkpoint.  The PANIC would be
caused by the redo LSN referred in the control file as located in a
segment already gone, recycled by the previous restartpoint with
"checkPoint" out-of-sync in the control file.

This commit fixes the update of the control file during restartpoints so
as "checkPoint" and "checkPointCopy" are updated even if the cluster has
been promoted while a restartpoint is running, to be on par with the set
of WAL segments actually recycled in the end of CreateRestartPoint().

This problem exists in all the stable branches.  However, commit
7ff23c6, by removing the last call of CreateCheckPoint() from the
startup process, has made this bug much easier to reason about as
concurrent checkpoints are not possible anymore.  No backpatch is done
yet, mostly out of caution from me as a point release is close by, but
we need to think harder about the case of concurrent checkpoints at
promotion if the bgwriter is not considered as running by the startup
process in ~v14, so this change is done only on HEAD for the moment.

Reported-by: Fujii Masao, Rui Zhao
Author: Kyotaro Horiguchi
Reviewed-by: Nathan Bossart, Michael Paquier
Discussion: https://postgr.es/m/20220316.102444.2193181487576617583.horikyota.ntt@gmail.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/7863ee4def653f2c2193cb0b0cf4a8f0f3ca6c56

Modified Files
--------------
src/backend/access/transam/xlog.c | 54 ++++++++++++++++++++++++---------------
1 file changed, 33 insertions(+), 21 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* pgsql: Fix control file update done in restartpoints still running afte
@ 2022-05-16 02:28  Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2022-05-16 02:28 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix control file update done in restartpoints still running after promotion

If a cluster is promoted (aka the control file shows a state different
than DB_IN_ARCHIVE_RECOVERY) while CreateRestartPoint() is still
processing, this function could miss an update of the control file for
"checkPoint" and "checkPointCopy" but still do the recycling and/or
removal of the past WAL segments, assuming that the to-be-updated LSN
values should be used as reference points for the cleanup.  This causes
a follow-up restart attempting crash recovery to fail with a PANIC on a
missing checkpoint record if the end-of-recovery checkpoint triggered by
the promotion did not complete while the cluster abruptly stopped or
crashed before the completion of this checkpoint.  The PANIC would be
caused by the redo LSN referred in the control file as located in a
segment already gone, recycled by the previous restartpoint with
"checkPoint" out-of-sync in the control file.

This commit fixes the update of the control file during restartpoints so
as "checkPoint" and "checkPointCopy" are updated even if the cluster has
been promoted while a restartpoint is running, to be on par with the set
of WAL segments actually recycled in the end of CreateRestartPoint().

7863ee4 has fixed this problem already on master, but the release timing
of the latest point versions did not let me enough time to study and fix
that on all the stable branches.

Reported-by: Fujii Masao, Rui Zhao
Author: Kyotaro Horiguchi
Reviewed-by: Nathan Bossart, Michael Paquier
Discussion: https://postgr.es/m/20220316.102444.2193181487576617583.horikyota.ntt@gmail.com
Backpatch-through: 10

Branch
------
REL_14_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/6dced63b411b920f7d0296337e4b3c8cba3e2e30

Modified Files
--------------
src/backend/access/transam/xlog.c | 44 ++++++++++++++++++++++++++-------------
1 file changed, 29 insertions(+), 15 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* pgsql: Fix control file update done in restartpoints still running afte
@ 2022-05-16 02:28  Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2022-05-16 02:28 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix control file update done in restartpoints still running after promotion

If a cluster is promoted (aka the control file shows a state different
than DB_IN_ARCHIVE_RECOVERY) while CreateRestartPoint() is still
processing, this function could miss an update of the control file for
"checkPoint" and "checkPointCopy" but still do the recycling and/or
removal of the past WAL segments, assuming that the to-be-updated LSN
values should be used as reference points for the cleanup.  This causes
a follow-up restart attempting crash recovery to fail with a PANIC on a
missing checkpoint record if the end-of-recovery checkpoint triggered by
the promotion did not complete while the cluster abruptly stopped or
crashed before the completion of this checkpoint.  The PANIC would be
caused by the redo LSN referred in the control file as located in a
segment already gone, recycled by the previous restartpoint with
"checkPoint" out-of-sync in the control file.

This commit fixes the update of the control file during restartpoints so
as "checkPoint" and "checkPointCopy" are updated even if the cluster has
been promoted while a restartpoint is running, to be on par with the set
of WAL segments actually recycled in the end of CreateRestartPoint().

7863ee4 has fixed this problem already on master, but the release timing
of the latest point versions did not let me enough time to study and fix
that on all the stable branches.

Reported-by: Fujii Masao, Rui Zhao
Author: Kyotaro Horiguchi
Reviewed-by: Nathan Bossart, Michael Paquier
Discussion: https://postgr.es/m/20220316.102444.2193181487576617583.horikyota.ntt@gmail.com
Backpatch-through: 10

Branch
------
REL_11_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/4525151d4c2f32dfeeef817968172e4c3328fa52

Modified Files
--------------
src/backend/access/transam/xlog.c | 44 ++++++++++++++++++++++++++-------------
1 file changed, 29 insertions(+), 15 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* pgsql: Fix control file update done in restartpoints still running afte
@ 2022-05-16 02:28  Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2022-05-16 02:28 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix control file update done in restartpoints still running after promotion

If a cluster is promoted (aka the control file shows a state different
than DB_IN_ARCHIVE_RECOVERY) while CreateRestartPoint() is still
processing, this function could miss an update of the control file for
"checkPoint" and "checkPointCopy" but still do the recycling and/or
removal of the past WAL segments, assuming that the to-be-updated LSN
values should be used as reference points for the cleanup.  This causes
a follow-up restart attempting crash recovery to fail with a PANIC on a
missing checkpoint record if the end-of-recovery checkpoint triggered by
the promotion did not complete while the cluster abruptly stopped or
crashed before the completion of this checkpoint.  The PANIC would be
caused by the redo LSN referred in the control file as located in a
segment already gone, recycled by the previous restartpoint with
"checkPoint" out-of-sync in the control file.

This commit fixes the update of the control file during restartpoints so
as "checkPoint" and "checkPointCopy" are updated even if the cluster has
been promoted while a restartpoint is running, to be on par with the set
of WAL segments actually recycled in the end of CreateRestartPoint().

7863ee4 has fixed this problem already on master, but the release timing
of the latest point versions did not let me enough time to study and fix
that on all the stable branches.

Reported-by: Fujii Masao, Rui Zhao
Author: Kyotaro Horiguchi
Reviewed-by: Nathan Bossart, Michael Paquier
Discussion: https://postgr.es/m/20220316.102444.2193181487576617583.horikyota.ntt@gmail.com
Backpatch-through: 10

Branch
------
REL_12_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/7e59b1219109be1b42d1f3937d09637327a56f5b

Modified Files
--------------
src/backend/access/transam/xlog.c | 44 ++++++++++++++++++++++++++-------------
1 file changed, 29 insertions(+), 15 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* pgsql: Fix control file update done in restartpoints still running afte
@ 2022-05-16 02:28  Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2022-05-16 02:28 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix control file update done in restartpoints still running after promotion

If a cluster is promoted (aka the control file shows a state different
than DB_IN_ARCHIVE_RECOVERY) while CreateRestartPoint() is still
processing, this function could miss an update of the control file for
"checkPoint" and "checkPointCopy" but still do the recycling and/or
removal of the past WAL segments, assuming that the to-be-updated LSN
values should be used as reference points for the cleanup.  This causes
a follow-up restart attempting crash recovery to fail with a PANIC on a
missing checkpoint record if the end-of-recovery checkpoint triggered by
the promotion did not complete while the cluster abruptly stopped or
crashed before the completion of this checkpoint.  The PANIC would be
caused by the redo LSN referred in the control file as located in a
segment already gone, recycled by the previous restartpoint with
"checkPoint" out-of-sync in the control file.

This commit fixes the update of the control file during restartpoints so
as "checkPoint" and "checkPointCopy" are updated even if the cluster has
been promoted while a restartpoint is running, to be on par with the set
of WAL segments actually recycled in the end of CreateRestartPoint().

7863ee4 has fixed this problem already on master, but the release timing
of the latest point versions did not let me enough time to study and fix
that on all the stable branches.

Reported-by: Fujii Masao, Rui Zhao
Author: Kyotaro Horiguchi
Reviewed-by: Nathan Bossart, Michael Paquier
Discussion: https://postgr.es/m/20220316.102444.2193181487576617583.horikyota.ntt@gmail.com
Backpatch-through: 10

Branch
------
REL_13_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/2e9559b30239ce3cf69383ced208a72a7eb99335

Modified Files
--------------
src/backend/access/transam/xlog.c | 44 ++++++++++++++++++++++++++-------------
1 file changed, 29 insertions(+), 15 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread

* pgsql: Fix control file update done in restartpoints still running afte
@ 2022-05-16 02:28  Michael Paquier <michael@paquier.xyz>
  0 siblings, 0 replies; 6+ messages in thread

From: Michael Paquier @ 2022-05-16 02:28 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

Fix control file update done in restartpoints still running after promotion

If a cluster is promoted (aka the control file shows a state different
than DB_IN_ARCHIVE_RECOVERY) while CreateRestartPoint() is still
processing, this function could miss an update of the control file for
"checkPoint" and "checkPointCopy" but still do the recycling and/or
removal of the past WAL segments, assuming that the to-be-updated LSN
values should be used as reference points for the cleanup.  This causes
a follow-up restart attempting crash recovery to fail with a PANIC on a
missing checkpoint record if the end-of-recovery checkpoint triggered by
the promotion did not complete while the cluster abruptly stopped or
crashed before the completion of this checkpoint.  The PANIC would be
caused by the redo LSN referred in the control file as located in a
segment already gone, recycled by the previous restartpoint with
"checkPoint" out-of-sync in the control file.

This commit fixes the update of the control file during restartpoints so
as "checkPoint" and "checkPointCopy" are updated even if the cluster has
been promoted while a restartpoint is running, to be on par with the set
of WAL segments actually recycled in the end of CreateRestartPoint().

7863ee4 has fixed this problem already on master, but the release timing
of the latest point versions did not let me enough time to study and fix
that on all the stable branches.

Reported-by: Fujii Masao, Rui Zhao
Author: Kyotaro Horiguchi
Reviewed-by: Nathan Bossart, Michael Paquier
Discussion: https://postgr.es/m/20220316.102444.2193181487576617583.horikyota.ntt@gmail.com
Backpatch-through: 10

Branch
------
REL_10_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/60e956eb825c5e0eb72c17556f8e60993100f8c5

Modified Files
--------------
src/backend/access/transam/xlog.c | 44 ++++++++++++++++++++++++++-------------
1 file changed, 29 insertions(+), 15 deletions(-)



^ permalink  raw  reply  [nested|flat] 6+ messages in thread


end of thread, other threads:[~2022-05-16 02:28 UTC | newest]

Thread overview: 6+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-05-08 23:41 pgsql: Fix control file update done in restartpoints still running afte Michael Paquier <michael@paquier.xyz>
2022-05-16 02:28 pgsql: Fix control file update done in restartpoints still running afte Michael Paquier <michael@paquier.xyz>
2022-05-16 02:28 pgsql: Fix control file update done in restartpoints still running afte Michael Paquier <michael@paquier.xyz>
2022-05-16 02:28 pgsql: Fix control file update done in restartpoints still running afte Michael Paquier <michael@paquier.xyz>
2022-05-16 02:28 pgsql: Fix control file update done in restartpoints still running afte Michael Paquier <michael@paquier.xyz>
2022-05-16 02:28 pgsql: Fix control file update done in restartpoints still running afte Michael Paquier <michael@paquier.xyz>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox