public inbox for [email protected]
help / color / mirror / Atom feedFrom: Imseih (AWS), Sami <[email protected]>
To: Kyotaro Horiguchi <[email protected]>
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: [BUG] Panic due to incorrect missingContrecPtr after promotion
Date: Mon, 7 Mar 2022 20:34:12 +0000
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
I have gone ahead and backpatched this all the way to 10 as well.
--
Sami Imseih
Amazon Web Services
Attachments:
[application/octet-stream] v4-0001-Fix-missing-continuation-record-after-standby-promot-v10.patch (2.5K, ../[email protected]/2-v4-0001-Fix-missing-continuation-record-after-standby-promot-v10.patch)
download | inline diff:
From ae10aaed56ba309bcb8a32d74a7f5fc9ea466c5c Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Mon, 7 Mar 2022 19:43:50 +0000
Subject: [PATCH 1/1] Fix "missing continuation record" after standby promotion
Invalidate abortedRecPtr and missingContrecPtr after a missing
continuation record is skipped on a standby. This fixes a PANIC
caused when a recently promoted standby attempts to write an
OVERWRITE_RECORD with an LSN of the previously read aborted record.
Author: Sami Imseih
Reviewed-by: Kyotaro Horiguchi <[email protected]>, Alvaro Herrera <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Backpatch to 10
---
src/backend/access/transam/xlog.c | 4 ++++
src/test/recovery/t/026_overwrite_contrecord.pl | 8 +++++++-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index c64febdb53..76fe62af33 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10260,6 +10260,10 @@ VerifyOverwriteContrecord(xl_overwrite_contrecord *xlrec, XLogReaderState *state
(uint32) (state->overwrittenRecPtr >> 32),
(uint32) state->overwrittenRecPtr);
+ /* We have safely skipped the aborted record */
+ abortedRecPtr = InvalidXLogRecPtr;
+ missingContrecPtr = InvalidXLogRecPtr;
+
ereport(LOG,
(errmsg("successfully skipped missing contrecord at %X/%X, overwritten at %s",
(uint32) (xlrec->overwritten_lsn >> 32),
diff --git a/src/test/recovery/t/026_overwrite_contrecord.pl b/src/test/recovery/t/026_overwrite_contrecord.pl
index 57b2a6b7fb..ef962b1271 100644
--- a/src/test/recovery/t/026_overwrite_contrecord.pl
+++ b/src/test/recovery/t/026_overwrite_contrecord.pl
@@ -15,7 +15,7 @@ plan tests => 3;
# Test: Create a physical replica that's missing the last WAL file,
# then restart the primary to create a divergent WAL file and observe
# that the replica replays the "overwrite contrecord" from that new
-# file.
+# file and the standby promotes successfully.
my $node = PostgresNode->get_new_node('primary');
$node->init(allows_streaming => 1);
@@ -105,5 +105,11 @@ like(
qr[successfully skipped missing contrecord at],
"found log line in standby");
+# Verify promotion is successful
+$ENV{PGDATA} = $node_standby->data_dir;
+$ENV{PGPORT} = $node_standby->port;
+$ENV{PGGHOST} = $node_standby->host;
+system "pg_ctl promote";
+
$node->stop;
$node_standby->stop;
--
2.32.0
[application/octet-stream] v4-0001-Fix-missing-continuation-record-after-standby-promot-v11.patch (2.5K, ../[email protected]/3-v4-0001-Fix-missing-continuation-record-after-standby-promot-v11.patch)
download | inline diff:
From 6fab9a83c7a6fce4f2fdd748697f23921a5df2ed Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Mon, 7 Mar 2022 19:49:32 +0000
Subject: [PATCH 1/1] Fix "missing continuation record" after standby promotion
Invalidate abortedRecPtr and missingContrecPtr after a missing
continuation record is skipped on a standby. This fixes a PANIC
caused when a recently promoted standby attempts to write an
OVERWRITE_RECORD with an LSN of the previously read aborted record.
Author: Sami Imseih
Reviewed-by: Kyotaro Horiguchi <[email protected]>, Alvaro Herrera <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Backpatch to 10
---
src/backend/access/transam/xlog.c | 4 ++++
src/test/recovery/t/026_overwrite_contrecord.pl | 8 +++++++-
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 12992f5f3d..5afb3a5f1d 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10296,6 +10296,10 @@ VerifyOverwriteContrecord(xl_overwrite_contrecord *xlrec, XLogReaderState *state
(uint32) (state->overwrittenRecPtr >> 32),
(uint32) state->overwrittenRecPtr);
+ /* We have safely skipped the aborted record */
+ abortedRecPtr = InvalidXLogRecPtr;
+ missingContrecPtr = InvalidXLogRecPtr;
+
ereport(LOG,
(errmsg("successfully skipped missing contrecord at %X/%X, overwritten at %s",
(uint32) (xlrec->overwritten_lsn >> 32),
diff --git a/src/test/recovery/t/026_overwrite_contrecord.pl b/src/test/recovery/t/026_overwrite_contrecord.pl
index 867b75937e..f51547e149 100644
--- a/src/test/recovery/t/026_overwrite_contrecord.pl
+++ b/src/test/recovery/t/026_overwrite_contrecord.pl
@@ -15,7 +15,7 @@ plan tests => 3;
# Test: Create a physical replica that's missing the last WAL file,
# then restart the primary to create a divergent WAL file and observe
# that the replica replays the "overwrite contrecord" from that new
-# file.
+# file and the standby promotes successfully.
my $node = PostgresNode->get_new_node('primary');
$node->init(allows_streaming => 1);
@@ -102,5 +102,11 @@ like(
qr[successfully skipped missing contrecord at],
"found log line in standby");
+# Verify promotion is successful
+$ENV{PGDATA} = $node_standby->data_dir;
+$ENV{PGPORT} = $node_standby->port;
+$ENV{PGGHOST} = $node_standby->host;
+system "pg_ctl promote";
+
$node->stop;
$node_standby->stop;
--
2.32.0
[application/octet-stream] v4-0001-Fix-missing-continuation-record-after-standby-promot-v12.patch (2.4K, ../[email protected]/4-v4-0001-Fix-missing-continuation-record-after-standby-promot-v12.patch)
download | inline diff:
From 4daed94b2c32c1eb11cf37e87d11ea2c5ae803cd Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Fri, 4 Mar 2022 19:51:15 +0000
Subject: [PATCH 1/1] Fix "missing continuation record" after standby promotion
Invalidate abortedRecPtr and missingContrecPtr after a missing
continuation record is skipped on a standby. This fixes a PANIC
caused when a recently promoted standby attempts to write an
OVERWRITE_RECORD with an LSN of the previously read aborted record.
Author: Sami Imseih
Reviewed-by: Kyotaro Horiguchi <[email protected]>, Alvaro Herrera <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Backpatch to 10
---
src/backend/access/transam/xlog.c | 4 ++++
src/test/recovery/t/026_overwrite_contrecord.pl | 5 ++++-
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 885558f291..8c7bb1dc6f 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10148,6 +10148,10 @@ VerifyOverwriteContrecord(xl_overwrite_contrecord *xlrec, XLogReaderState *state
(uint32) (state->overwrittenRecPtr >> 32),
(uint32) state->overwrittenRecPtr);
+ /* We have safely skipped the aborted record */
+ abortedRecPtr = InvalidXLogRecPtr;
+ missingContrecPtr = InvalidXLogRecPtr;
+
ereport(LOG,
(errmsg("successfully skipped missing contrecord at %X/%X, overwritten at %s",
(uint32) (xlrec->overwritten_lsn >> 32),
diff --git a/src/test/recovery/t/026_overwrite_contrecord.pl b/src/test/recovery/t/026_overwrite_contrecord.pl
index 867b75937e..9115b5ffc9 100644
--- a/src/test/recovery/t/026_overwrite_contrecord.pl
+++ b/src/test/recovery/t/026_overwrite_contrecord.pl
@@ -15,7 +15,7 @@ plan tests => 3;
# Test: Create a physical replica that's missing the last WAL file,
# then restart the primary to create a divergent WAL file and observe
# that the replica replays the "overwrite contrecord" from that new
-# file.
+# file and the standby promotes successfully.
my $node = PostgresNode->get_new_node('primary');
$node->init(allows_streaming => 1);
@@ -102,5 +102,8 @@ like(
qr[successfully skipped missing contrecord at],
"found log line in standby");
+# Verify promotion is successful
+$node_standby->safe_psql('postgres', "select pg_promote()");
+
$node->stop;
$node_standby->stop;
--
2.32.0
[application/octet-stream] v4-0001-Fix-missing-continuation-record-after-standby-promot-v13.patch (2.4K, ../[email protected]/5-v4-0001-Fix-missing-continuation-record-after-standby-promot-v13.patch)
download | inline diff:
From 79fb2465daf73bf9f710e55860f68df755d614b8 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Fri, 4 Mar 2022 23:57:02 +0000
Subject: [PATCH 1/1] Fix "missing continuation record" after standby promotion
Invalidate abortedRecPtr and missingContrecPtr after a missing
continuation record is skipped on a standby. This fixes a PANIC
caused when a recently promoted standby attempts to write an
OVERWRITE_RECORD with an LSN of the previously read aborted record.
Author: Sami Imseih
Reviewed-by: Kyotaro Horiguchi <[email protected]>, Alvaro Herrera <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Backpatch to 10
---
src/backend/access/transam/xlog.c | 4 ++++
src/test/recovery/t/026_overwrite_contrecord.pl | 5 ++++-
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 3d76fad128..a472e26328 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10382,6 +10382,10 @@ VerifyOverwriteContrecord(xl_overwrite_contrecord *xlrec, XLogReaderState *state
(uint32) (state->overwrittenRecPtr >> 32),
(uint32) state->overwrittenRecPtr);
+ /* We have safely skipped the aborted record */
+ abortedRecPtr = InvalidXLogRecPtr;
+ missingContrecPtr = InvalidXLogRecPtr;
+
ereport(LOG,
(errmsg("successfully skipped missing contrecord at %X/%X, overwritten at %s",
(uint32) (xlrec->overwritten_lsn >> 32),
diff --git a/src/test/recovery/t/026_overwrite_contrecord.pl b/src/test/recovery/t/026_overwrite_contrecord.pl
index cb7f5b68c6..2577afc98b 100644
--- a/src/test/recovery/t/026_overwrite_contrecord.pl
+++ b/src/test/recovery/t/026_overwrite_contrecord.pl
@@ -15,7 +15,7 @@ plan tests => 3;
# Test: Create a physical replica that's missing the last WAL file,
# then restart the primary to create a divergent WAL file and observe
# that the replica replays the "overwrite contrecord" from that new
-# file.
+# file and the standby promotes successfully.
my $node = PostgresNode->get_new_node('primary');
$node->init(allows_streaming => 1);
@@ -102,5 +102,8 @@ like(
qr[successfully skipped missing contrecord at],
"found log line in standby");
+# Verify promotion is successful
+$node_standby->safe_psql('postgres', "select pg_promote()");
+
$node->stop;
$node_standby->stop;
--
2.32.0
[application/octet-stream] v4-0001-Fix-missing-continuation-record-after-standby-promot-v14.patch (2.4K, ../[email protected]/6-v4-0001-Fix-missing-continuation-record-after-standby-promot-v14.patch)
download | inline diff:
From 8332d475825f05f5c9f7f48537d4c5cbbc6f29c4 Mon Sep 17 00:00:00 2001
From: "Sami Imseih (AWS)" <[email protected]>
Date: Fri, 4 Mar 2022 19:10:18 +0000
Subject: [PATCH 1/1] Fix "missing continuation record" after standby promotion
Invalidate abortedRecPtr and missingContrecPtr after a missing
continuation record is skipped on a standby. This fixes a PANIC
caused when a recently promoted standby attempts to write an
OVERWRITE_RECORD with an LSN of the previously read aborted record.
Author: Sami Imseih
Reviewed-by: Kyotaro Horiguchi <[email protected]>, Alvaro Herrera <[email protected]>
Discussion: https://postgr.es/m/[email protected]
Backpatch to 10
---
src/backend/access/transam/xlog.c | 4 ++++
src/test/recovery/t/026_overwrite_contrecord.pl | 5 ++++-
2 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 6208e123e5..b241f33f7e 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -10591,6 +10591,10 @@ VerifyOverwriteContrecord(xl_overwrite_contrecord *xlrec, XLogReaderState *state
LSN_FORMAT_ARGS(xlrec->overwritten_lsn),
LSN_FORMAT_ARGS(state->overwrittenRecPtr));
+ /* We have safely skipped the aborted record */
+ abortedRecPtr = InvalidXLogRecPtr;
+ missingContrecPtr = InvalidXLogRecPtr;
+
ereport(LOG,
(errmsg("successfully skipped missing contrecord at %X/%X, overwritten at %s",
LSN_FORMAT_ARGS(xlrec->overwritten_lsn),
diff --git a/src/test/recovery/t/026_overwrite_contrecord.pl b/src/test/recovery/t/026_overwrite_contrecord.pl
index cb7f5b68c6..2577afc98b 100644
--- a/src/test/recovery/t/026_overwrite_contrecord.pl
+++ b/src/test/recovery/t/026_overwrite_contrecord.pl
@@ -15,7 +15,7 @@ plan tests => 3;
# Test: Create a physical replica that's missing the last WAL file,
# then restart the primary to create a divergent WAL file and observe
# that the replica replays the "overwrite contrecord" from that new
-# file.
+# file and the standby promotes successfully.
my $node = PostgresNode->get_new_node('primary');
$node->init(allows_streaming => 1);
@@ -102,5 +102,8 @@ like(
qr[successfully skipped missing contrecord at],
"found log line in standby");
+# Verify promotion is successful
+$node_standby->safe_psql('postgres', "select pg_promote()");
+
$node->stop;
$node_standby->stop;
--
2.32.0
view thread (11+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: [BUG] Panic due to incorrect missingContrecPtr after promotion
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox