agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH v11 5/7] Row pattern recognition patch (docs). 267+ messages / 2 participants [nested] [flat]
* [PATCH v11 5/7] Row pattern recognition patch (docs). @ 2023-11-08 06:57 Tatsuo Ishii <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Tatsuo Ishii @ 2023-11-08 06:57 UTC (permalink / raw) --- doc/src/sgml/advanced.sgml | 80 ++++++++++++++++++++++++++++++++++++ doc/src/sgml/func.sgml | 54 ++++++++++++++++++++++++ doc/src/sgml/ref/select.sgml | 38 ++++++++++++++++- 3 files changed, 170 insertions(+), 2 deletions(-) diff --git a/doc/src/sgml/advanced.sgml b/doc/src/sgml/advanced.sgml index 755c9f1485..cf18dd887e 100644 --- a/doc/src/sgml/advanced.sgml +++ b/doc/src/sgml/advanced.sgml @@ -537,6 +537,86 @@ WHERE pos < 3; <literal>rank</literal> less than 3. </para> + <para> + Row pattern common syntax can be used to perform row pattern recognition + in a query. Row pattern common syntax includes two sub + clauses: <literal>DEFINE</literal> + and <literal>PATTERN</literal>. <literal>DEFINE</literal> defines + definition variables along with an expression. The expression must be a + logical expression, which means it must + return <literal>TRUE</literal>, <literal>FALSE</literal> + or <literal>NULL</literal>. The expression may comprise column references + and functions. Window functions, aggregate functions and subqueries are + not allowed. An example of <literal>DEFINE</literal> is as follows. + +<programlisting> +DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +</programlisting> + + Note that <function>PREV</function> returns the price column in the + previous row if it's called in a context of row pattern recognition. So in + the second line the definition variable "UP" is <literal>TRUE</literal> + when the price column in the current row is greater than the price column + in the previous row. Likewise, "DOWN" is <literal>TRUE</literal> when when + the price column in the current row is lower than the price column in the + previous row. + </para> + <para> + Once <literal>DEFINE</literal> exists, <literal>PATTERN</literal> can be + used. <literal>PATTERN</literal> defines a sequence of rows that satisfies + certain conditions. For example following <literal>PATTERN</literal> + defines that a row starts with the condition "LOWPRICE", then one or more + rows satisfy "UP" and finally one or more rows satisfy "DOWN". Note that + "+" means one or more matches. Also you can use "*", which means zero or + more matches. If a sequence of rows which satisfies the PATTERN is found, + in the starting row of the sequence of rows all window functions and + aggregates are shown in the target list. Note that aggregations only look + into the matched rows, rather than whole frame. In the second or + subsequent rows all window functions and aggregates are NULL. For rows + that do not match the PATTERN, all window functions and aggregates are + shown AS NULL too, except count which shows 0. This is because the + unmatched rows are in an empty frame. Example of + a <literal>SELECT</literal> using the <literal>DEFINE</literal> + and <literal>PATTERN</literal> clause is as follows. + +<programlisting> +SELECT company, tdate, price, + first_value(price) OVER w, + max(price) OVER w, + count(price) OVER w +FROM stock, + WINDOW w AS ( + PARTITION BY company + ORDER BY tdate + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING + AFTER MATCH SKIP PAST LAST ROW + INITIAL + PATTERN (LOWPRICE UP+ DOWN+) + DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +); +</programlisting> +<screen> + company | tdate | price | first_value | max | count +----------+------------+-------+-------------+-----+------- + company1 | 2023-07-01 | 100 | 100 | 200 | 4 + company1 | 2023-07-02 | 200 | | | + company1 | 2023-07-03 | 150 | | | + company1 | 2023-07-04 | 140 | | | + company1 | 2023-07-05 | 150 | | | 0 + company1 | 2023-07-06 | 90 | 90 | 130 | 4 + company1 | 2023-07-07 | 110 | | | + company1 | 2023-07-08 | 130 | | | + company1 | 2023-07-09 | 120 | | | + company1 | 2023-07-10 | 130 | | | 0 +</screen> + </para> + <para> When a query involves multiple window functions, it is possible to write out each one with a separate <literal>OVER</literal> clause, but this is diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index d963f0a0a0..c3a8167c8e 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -21933,6 +21933,7 @@ SELECT count(*) FROM sometable; returns <literal>NULL</literal> if there is no such row. </para></entry> </row> + </tbody> </tgroup> </table> @@ -21972,6 +21973,59 @@ SELECT count(*) FROM sometable; Other frame specifications can be used to obtain other effects. </para> + <para> + Row pattern recognition navigation functions are listed in + <xref linkend="functions-rpr-navigation-table"/>. These functions + can be used to describe DEFINE clause of Row pattern recognition. + </para> + + <table id="functions-rpr-navigation-table"> + <title>Row Pattern Navigation Functions</title> + <tgroup cols="1"> + <thead> + <row> + <entry role="func_table_entry"><para role="func_signature"> + Function + </para> + <para> + Description + </para></entry> + </row> + </thead> + + <tbody> + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>prev</primary> + </indexterm> + <function>prev</function> ( <parameter>value</parameter> <type>anyelement</type> ) + <returnvalue>anyelement</returnvalue> + </para> + <para> + Returns the column value at the previous row; + returns NULL if there is no previous row in the window frame. + </para></entry> + </row> + + <row> + <entry role="func_table_entry"><para role="func_signature"> + <indexterm> + <primary>next</primary> + </indexterm> + <function>next</function> ( <parameter>value</parameter> <type>anyelement</type> ) + <returnvalue>anyelement</returnvalue> + </para> + <para> + Returns the column value at the next row; + returns NULL if there is no next row in the window frame. + </para></entry> + </row> + + </tbody> + </tgroup> + </table> + <note> <para> The SQL standard defines a <literal>RESPECT NULLS</literal> or diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml index 42d78913cf..522ad9dd70 100644 --- a/doc/src/sgml/ref/select.sgml +++ b/doc/src/sgml/ref/select.sgml @@ -969,8 +969,8 @@ WINDOW <replaceable class="parameter">window_name</replaceable> AS ( <replaceabl The <replaceable class="parameter">frame_clause</replaceable> can be one of <synopsis> -{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] -{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] +{ RANGE | ROWS | GROUPS } <replaceable>frame_start</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax] +{ RANGE | ROWS | GROUPS } BETWEEN <replaceable>frame_start</replaceable> AND <replaceable>frame_end</replaceable> [ <replaceable>frame_exclusion</replaceable> ] [row_pattern_common_syntax] </synopsis> where <replaceable>frame_start</replaceable> @@ -1077,6 +1077,40 @@ EXCLUDE NO OTHERS a given peer group will be in the frame or excluded from it. </para> + <para> + The + optional <replaceable class="parameter">row_pattern_common_syntax</replaceable> + defines the <firstterm>row pattern recognition condition</firstterm> for + this + window. <replaceable class="parameter">row_pattern_common_syntax</replaceable> + includes following subclauses. <literal>AFTER MATCH SKIP PAST LAST + ROW</literal> or <literal>AFTER MATCH SKIP TO NEXT ROW</literal> controls + how to proceed to next row position after a match + found. With <literal>AFTER MATCH SKIP PAST LAST ROW</literal> (the + default) next row position is next to the last row of previous match. On + the other hand, with <literal>AFTER MATCH SKIP TO NEXT ROW</literal> next + row position is always next to the last row of previous + match. <literal>DEFINE</literal> defines definition variables along with a + boolean expression. <literal>PATTERN</literal> defines a sequence of rows + that satisfies certain conditions using variables defined + in <literal>DEFINE</literal> clause. If the variable is not defined in + the <literal>DEFINE</literal> clause, it is implicitly assumed + following is defined in the <literal>DEFINE</literal> clause. + +<synopsis> +<literal>variable_name</literal> AS TRUE +</synopsis> + + Note that the maximu number of variables defined + in <literal>DEFINE</literal> clause is 26. + +<synopsis> +[ AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW ] +PATTERN <replaceable class="parameter">pattern_variable_name</replaceable>[+] [, ...] +DEFINE <replaceable class="parameter">definition_varible_name</replaceable> AS <replaceable class="parameter">expression</replaceable> [, ...] +</synopsis> + </para> + <para> The purpose of a <literal>WINDOW</literal> clause is to specify the behavior of <firstterm>window functions</firstterm> appearing in the query's -- 2.25.1 ----Next_Part(Wed_Nov__8_16_37_05_2023_872)-- Content-Type: Text/X-Patch; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="v11-0006-Row-pattern-recognition-patch-tests.patch" ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
* [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. @ 2024-05-18 01:41 Thomas Munro <[email protected]> 0 siblings, 0 replies; 267+ messages in thread From: Thomas Munro @ 2024-05-18 01:41 UTC (permalink / raw) When backend processes were launched in EXEC_BACKEND builds, they would run LocalProcessControlFile() to read in pg_control and extract several important settings. This happens too early to acquire ControlFileLock, and the postmaster is also not allowed to acquire ControlFileLock, so it can't safely take a copy to give to the child. Instead, pass down the "proto-controlfile" that was read by the postmaster in LocalProcessControlFile(). Introduce functions ExportProtoControlFile() and ImportProtoControlFile() to allow that. Subprocesses will extract information from that, and then later attach to the current control file in shared memory. Reported-by: Melanie Plageman <[email protected]> per Windows CI failure Discussion: https://postgr.es/m/CAAKRu_YNGwEYrorQYza_W8tU%2B%3DtoXRHG8HpyHC-KDbZqA_ZVSA%40mail.gmail.com --- src/backend/access/transam/xlog.c | 46 +++++++++++++++++++++++-- src/backend/postmaster/launch_backend.c | 21 +++++++---- src/include/access/xlog.h | 5 +++ 3 files changed, 64 insertions(+), 8 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 13ec6225b85..e52517eb9c1 100644 --- a/src/backend/access/transam/xlog.c +++ b/src/backend/access/transam/xlog.c @@ -575,6 +575,10 @@ static WALInsertLockPadded *WALInsertLocks = NULL; */ static ControlFileData *ControlFile = NULL; +#ifdef EXEC_BACKEND +static ControlFileData *ProtoControlFile = NULL; +#endif + /* * Calculate the amount of space left on the page after 'endptr'. Beware * multiple evaluation! @@ -692,6 +696,7 @@ static bool PerformRecoveryXLogAction(void); static void InitControlFile(uint64 sysidentifier, uint32 data_checksum_version); static void WriteControlFile(void); static void ReadControlFile(void); +static void ScanControlFile(void); static void UpdateControlFile(void); static char *str_time(pg_time_t tnow, char *buf, size_t bufsize); @@ -4385,9 +4390,7 @@ WriteControlFile(void) static void ReadControlFile(void) { - pg_crc32c crc; int fd; - char wal_segsz_str[20]; int r; /* @@ -4420,6 +4423,15 @@ ReadControlFile(void) close(fd); + ScanControlFile(); +} + +static void +ScanControlFile(void) +{ + static char wal_segsz_str[20]; + pg_crc32c crc; + /* * Check for expected pg_control format version. If this is wrong, the * CRC check will likely fail because we'll be checking the wrong number @@ -4941,8 +4953,33 @@ LocalProcessControlFile(bool reset) Assert(reset || ControlFile == NULL); ControlFile = palloc_object(ControlFileData); ReadControlFile(); + +#ifdef EXEC_BACKEND + /* We need to be able to give this to subprocesses. */ + ProtoControlFile = ControlFile; +#endif } +#ifdef EXEC_BACKEND +void +ExportProtoControlFile(ControlFileData *copy) +{ + *copy = *ProtoControlFile; +} + +/* + * Like LocalProcessControlFile(), but used early in EXEC_BACKEND children's + * startup. This receives the same file that the postmaster first read. + */ +void +ImportProtoControlFile(const ControlFileData *copy) +{ + ControlFile = palloc(sizeof(ControlFileData)); + *ControlFile = *copy; + ScanControlFile(); +} +#endif + /* * Get the wal_level from the control file. For a standby, this value should be * considered as its active wal_level, because it may be different from what @@ -5061,7 +5098,12 @@ XLOGShmemInit(void) if (localControlFile) { memcpy(ControlFile, localControlFile, sizeof(ControlFileData)); +#ifdef EXEC_BACKEND + /* We still hold a reference to give to subprocesses. */ + Assert(ProtoControlFile == localControlFile); +#else pfree(localControlFile); +#endif } /* diff --git a/src/backend/postmaster/launch_backend.c b/src/backend/postmaster/launch_backend.c index 45690b11c99..e08a405f949 100644 --- a/src/backend/postmaster/launch_backend.c +++ b/src/backend/postmaster/launch_backend.c @@ -33,6 +33,9 @@ #include <unistd.h> +#include "access/xlog.h" +#include "catalog/pg_control.h" +#include "common/file_utils.h" #include "libpq/libpq-be.h" #include "miscadmin.h" #include "postmaster/autovacuum.h" @@ -133,6 +136,14 @@ typedef struct int MyPMChildSlot; + /* + * A copy of the ControlFileData from early in Postmaster startup. We + * need to access its contents it at a phase of initialization before we + * are allowed to acquire LWLocks, so we can't just use shared memory or + * read the file from disk. + */ + ControlFileData proto_controlfile; + /* * These are only used by backend processes, but are here because passing * a socket needs some special handling on Windows. 'client_sock' is an @@ -659,12 +670,6 @@ SubPostmasterMain(int argc, char *argv[]) */ checkDataDir(); - /* - * (re-)read control file, as it contains config. The postmaster will - * already have read this, but this process doesn't know about that. - */ - LocalProcessControlFile(false); - /* * Reload any libraries that were preloaded by the postmaster. Since we * exec'd this process, those libraries didn't come along with us; but we @@ -752,6 +757,8 @@ save_backend_variables(BackendParameters *param, param->MaxBackends = MaxBackends; param->num_pmchild_slots = num_pmchild_slots; + ExportProtoControlFile(¶m->proto_controlfile); + #ifdef WIN32 param->PostmasterHandle = PostmasterHandle; if (!write_duplicated_handle(¶m->initial_signal_pipe, @@ -1026,6 +1033,8 @@ restore_backend_variables(BackendParameters *param) strlcpy(pkglib_path, param->pkglib_path, MAXPGPATH); + ImportProtoControlFile(¶m->proto_controlfile); + /* * We need to restore fd.c's counts of externally-opened FDs; to avoid * confusion, be sure to do this after restoring max_safe_fds. (Note: diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h index fdfb572467b..f1f3ad4e96e 100644 --- a/src/include/access/xlog.h +++ b/src/include/access/xlog.h @@ -208,6 +208,7 @@ typedef enum WALAvailability struct XLogRecData; struct XLogReaderState; +struct ControlFileData; extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata, XLogRecPtr fpw_lsn, @@ -250,6 +251,10 @@ extern void XLOGShmemInit(void); extern void BootStrapXLOG(uint32 data_checksum_version); extern void InitializeWalConsistencyChecking(void); extern void LocalProcessControlFile(bool reset); +#ifdef EXEC_BACKEND +extern void ExportProtoControlFile(struct ControlFileData *copy); +extern void ImportProtoControlFile(const struct ControlFileData *copy); +#endif extern WalLevel GetActiveWalLevelOnStandby(void); extern void StartupXLOG(void); extern void ShutdownXLOG(int code, Datum arg); -- 2.47.3 --dhbc6bswyy6qufwn-- ^ permalink raw reply [nested|flat] 267+ messages in thread
end of thread, other threads:[~2024-05-18 01:41 UTC | newest] Thread overview: 267+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2023-11-08 06:57 [PATCH v11 5/7] Row pattern recognition patch (docs). Tatsuo Ishii <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]> 2024-05-18 01:41 [PATCH v2] Fix pg_control corruption in EXEC_BACKEND startup. Thomas Munro <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox