agora inbox for [email protected]  

help / color / mirror / Atom feed

RE: [HACKERS] Re: SSL patch
267+ messages / 2 participants
[nested] [flat]

* RE: [HACKERS] Re: SSL patch
@ 1999-07-24 18:34  Magnus Hagander <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Magnus Hagander @ 1999-07-24 18:34 UTC (permalink / raw)
  To: 'Tom Lane' <[email protected]>; +Cc: pgsql-hackers

> >> Actually, it shouldn't matter whether the server is 6.6-without-SSL
> >> or pre-6.6.  At least in the way I envisioned it, they'd act the same.
> 
> > The 6.6-without-SSL still knows about the NEGOTIATE_SSL_CODE packet that
is
> > sent, and can respond "No, I can't do SSL". The pre-6.6 does not know
about
> > the existance of this packet, and will thus respond with "Unsupported
> > Frontend Protocol" (since it's a normal StartupPacket with the version
> > number set to something very large (like the cancel request was
> > implemented)).
> 
> OK, the point being that then the client can either break the connection
> (if it doesn't want to do an insecure connection) or send a
> StartupPacket to continue with an insecure connection.  I agree this
> will be a little quicker than tearing down the connection and starting
> a new one, which is what must happen in the case of an old server.
> 
> But you could save some code on both sides if you just made the
> teardown/new connection behavior the only path for falling back to
> non-SSL.  I'm not sure that SSL-enabled-client-talking-to-6.6-but-
> not-SSL-enabled-server will be a sufficiently common scenario to
> justify a lot of extra code to make it a tad faster.  You'd expect
> that most installations will have SSL at the server if they have
> it anywhere.
> 
> If it's only a small amount of extra code then it doesn't matter,
> of course.  I'm just dubious that it's worth taking extra care
> for that case when you are going to have the other case anyway.

Well. There is almost no extra code. The code has to be in place anyway, to
deal with those clients that do support SSL. It's just a matter of the
server sending a single character (a 'N' to say normal, non-SSL connection)
to the client after receiving the special packet that it knows about. It's
just:
#ifdef USE_SSL
	SSLok = 'S';
#else
	SSLok = 'N';
#endif

SSLok is then sent to the client. This code is only executed if the client
first asked for SSL - if it's a non-SSL client, this whole phase is skipped.
All the rest of the code is shared between the SSL and the non-SSL case.


I'll post a patch to the patches-list that makes the client work with a
pre-6.6 server in a minute. The noticable thing being that each time this
client tries to connect, the server logs an "Unsupported frontent protocol"
message. The client will then tear down it, and the connectDB() function
will call itself while being instructed to run with the pre-6.6 protocol. It
works fine for me to connect to a 6.4.2 server.


//Magnus



^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread

* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup.
@ 2024-05-18 01:41  Thomas Munro <[email protected]>
  0 siblings, 0 replies; 267+ messages in thread

From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw)

When backend processes were launched in EXEC_BACKEND builds, they would
run LocalProcessControlFile() to read in pg_control and extract several
important settings.

This happens too early to acquire ControlFileLock, and the postmaster is
also not allowed to acquire ControlFileLock, so it can't safely take a
copy to give to the child.

Instead, pass down the "proto-controlfile" that was read by the
postmaster in LocalProcessControlFile().  Introduce functions
ExportProtoControlFile() and ImportProtoControlFile() to allow that.
Subprocesses will extract information from that, and then later attach
to the current control file in shared memory.

Reported-by: Melanie Plageman <[email protected]> per Windows CI failure
Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com
---
 src/backend/access/transam/xlog.c       | 46 +++++++++++++++++++++++--
 src/backend/postmaster/launch_backend.c | 21 +++++++----
 src/include/access/xlog.h               |  5 +++
 3 files changed, 64 insertions(+), 8 deletions(-)

diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 13ec6225b85..e52517eb9c1 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL;
  */
 static ControlFileData *ControlFile = NULL;
 
+#ifdef EXEC_BACKEND
+static ControlFileData *ProtoControlFile = NULL;
+#endif
+
 /*
  * Calculate the amount of space left on the page after 'endptr'. Beware
  * multiple evaluation!
@@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void);
 static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version);
 static void WriteControlFile(void);
 static void ReadControlFile(void);
+static void ScanControlFile(void);
 static void UpdateControlFile(void);
 static char *str_time(pg_time_t tnow, char *buf, size_t bufsize);
 
@@ -4385,9 +4390,7 @@ WriteControlFile(void)
 static void
 ReadControlFile(void)
 {
-	pg_crc32c	crc;
 	int			fd;
-	char		wal_segsz_str[20];
 	int			r;
 
 	/*
@@ -4420,6 +4423,15 @@ ReadControlFile(void)
 
 	close(fd);
 
+	ScanControlFile();
+}
+
+static void
+ScanControlFile(void)
+{
+	static char wal_segsz_str[20];
+	pg_crc32c	crc;
+
 	/*
 	 * Check for expected pg_control format version.  If this is wrong, the
 	 * CRC check will likely fail because we'll be checking the wrong number
@@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset)
 	Assert(reset || ControlFile == NULL);
 	ControlFile = palloc_object(ControlFileData);
 	ReadControlFile();
+
+#ifdef EXEC_BACKEND
+	/* We need to be able to give this to subprocesses. */
+	ProtoControlFile = ControlFile;
+#endif
 }
 
+#ifdef EXEC_BACKEND
+void
+ExportProtoControlFile(ControlFileData *copy)
+{
+	*copy = *ProtoControlFile;
+}
+
+/*
+ * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's
+ * startup.  This receives the same file that the postmaster first read.
+ */
+void
+ImportProtoControlFile(const ControlFileData *copy)
+{
+	ControlFile = palloc(sizeof(ControlFileData));
+	*ControlFile = *copy;
+	ScanControlFile();
+}
+#endif
+
 /*
  * Get the wal_level from the control file. For a standby, this value should be
  * considered as its active wal_level, because it may be different from what
@@ -5061,7 +5098,12 @@ XLOGShmemInit(void)
 	if (localControlFile)
 	{
 		memcpy(ControlFile, localControlFile, sizeof(ControlFileData));
+#ifdef EXEC_BACKEND
+		/* We still hold a reference to give to subprocesses. */
+		Assert(ProtoControlFile == localControlFile);
+#else
 		pfree(localControlFile);
+#endif
 	}
 
 	/*
diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c
index 45690b11c99..e08a405f949 100644
--- a/src/backend/postmaster/launch_backend.c
+++ b/src/backend/postmaster/launch_backend.c
@@ -33,6 +33,9 @@
 
 #include <unistd.h>
 
+#include "access/xlog.h"
+#include "catalog/pg_control.h"
+#include "common/file_utils.h"
 #include "libpq/libpq-be.h"
 #include "miscadmin.h"
 #include "postmaster/autovacuum.h"
@@ -133,6 +136,14 @@ typedef struct
 
 	int			MyPMChildSlot;
 
+	/*
+	 * A copy of the ControlFileData from early in Postmaster startup.  We
+	 * need to access its contents it at a phase of initialization before we
+	 * are allowed to acquire LWLocks, so we can't just use shared memory or
+	 * read the file from disk.
+	 */
+	ControlFileData proto_controlfile;
+
 	/*
 	 * These are only used by backend processes, but are here because passing
 	 * a socket needs some special handling on Windows. 'client_sock' is an
@@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[])
 	 */
 	checkDataDir();
 
-	/*
-	 * (re-)read control file, as it contains config. The postmaster will
-	 * already have read this, but this process doesn't know about that.
-	 */
-	LocalProcessControlFile(false);
-
 	/*
 	 * Reload any libraries that were preloaded by the postmaster.  Since we
 	 * exec'd this process, those libraries didn't come along with us; but we
@@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param,
 	param->MaxBackends = MaxBackends;
 	param->num_pmchild_slots = num_pmchild_slots;
 
+	ExportProtoControlFile(&param->proto_controlfile);
+
 #ifdef WIN32
 	param->PostmasterHandle = PostmasterHandle;
 	if (!write_duplicated_handle(&param->initial_signal_pipe,
@@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param)
 
 	strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH);
 
+	ImportProtoControlFile(&param->proto_controlfile);
+
 	/*
 	 * We need to restore fd.c's counts of externally-opened FDs; to avoid
 	 * confusion, be sure to do this after restoring max_safe_fds.  (Note:
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index fdfb572467b..f1f3ad4e96e 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -208,6 +208,7 @@ typedef enum WALAvailability
 
 struct XLogRecData;
 struct XLogReaderState;
+struct ControlFileData;
 
 extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
 								   XLogRecPtr fpw_lsn,
@@ -250,6 +251,10 @@ extern void XLOGShmemInit(void);
 extern void BootStrapXLOG(uint32 data_checksum_version);
 extern void InitializeWalConsistencyChecking(void);
 extern void LocalProcessControlFile(bool reset);
+#ifdef EXEC_BACKEND
+extern void ExportProtoControlFile(struct ControlFileData *copy);
+extern void ImportProtoControlFile(const struct ControlFileData *copy);
+#endif
 extern WalLevel GetActiveWalLevelOnStandby(void);
 extern void StartupXLOG(void);
 extern void ShutdownXLOG(int code, Datum arg);
-- 
2.47.3


--dhbc6bswyy6qufwn--





^ permalink  raw  reply  [nested|flat] 267+ messages in thread